Sentry Backend Bugs

Sentry Backend Bug Pattern Review

Find bugs in Sentry backend code by checking for the patterns that cause the most real production errors.

This skill encodes patterns from 638 real production issues (393 resolved, 220 unresolved, 25 ignored) generating over 27 million error events across 65,000+ affected users. These are not theoretical risks -- they are the actual bugs that ship most often, with known fixes from resolved issues.

Scope

Review the code provided by the user, Warden, or the current branch diff. If the user does not provide a target, review the current branch diff. Start from the changed hunk or file, then read outward only as needed to confirm the behavior.

1. Analyze the changed code against the pattern checks below. 2. Use Read and Grep to trace data flow beyond the initial diff when needed. Follow function calls, callers, serializers, tasks, and ORM boundaries until the behavior is confirmed. 3. Report only HIGH and MEDIUM confidence findings.

Step 1: Classify the Code

Determine what you are reviewing and load the relevant reference.

If the code spans multiple categories, load all relevant references.

Step 2: Check for Top Bug Patterns

These are ordered by combined frequency and impact from real production data.

Check 1: Metric Subscription Query Errors -- 113 issues, 3,035,640 events

Alert and metric subscriptions referencing tags or functions that do not exist in the target dataset. These fire continuously once created.

Red flags:

• Creating Snuba subscriptions with SubscriptionData using user-provided query strings without validation
• Referencing transaction.duration in p95/p99 functions on the metrics dataset (it is a string type there)
• Using custom tag names (e.g., customerType) as filter dimensions without checking they exist
• Calling resolve_apdex_function without verifying the dataset supports threshold parameters

Safe patterns:

• Validate query fields against dataset schema before subscription creation
• Wrap _create_in_snuba calls with try/except SubscriptionError and mark subscription as invalid
• Use IncompatibleMetricsQuery checks before building metric subscription queries

Check 2: Missing Record / Stale Reference -- 81 issues, 1,403,592 events

Code calls .get() on a Django model assuming the record exists, but it has been deleted, merged, or never created.

Red flags:

• Model.objects.get(id=some_id) without try/except for DoesNotExist
• Detector.objects.get(id=detector_id) in workflow engine without handling deletion
• Environment.objects.get(name=env_name) in monitor/cron consumers
• Subscription.objects.get(id=sub_id) in billing tasks
• Using Group.objects.get() with IDs from Snuba query results (groups may be deleted/merged)
• Chained lookups where second .get() fails

Safe patterns:

• Model.objects.filter(...).first() with a None check
• try/except DoesNotExist that returns a graceful fallback (404, skip, log)
• Queryset .exists() check before .get()
• In API endpoints: return 404 for DoesNotExist, 400 for validation errors. Never suggest returning 500 intentionally.

Not a bug — do not flag:

• Infrastructure invariants: .get() enforcing a deployment precondition (e.g., "default org must exist in single-org mode") should crash — a 500 signals misconfiguration, not a code defect.
• Already validated by parent: If the endpoint base class validates the object (e.g., OrganizationEndpoint resolves the org), don't flag .get() on related records unless there's a genuine race or deletion window. Read the endpoint's parent class before reporting.
• Configuration lookups: Code that loads required config objects (get_default(), settings-based lookups) is expected to fail hard if the config is wrong.

Check 3: Search Query Validation -- 57 issues, 2,001,330 events

InvalidSearchQuery from user-provided or subscription-stored query filters referencing invalid values, deleted issues, or unresolved tags.

Red flags:

• by_qualified_short_id_bulk() called with short IDs from stored subscriptions that reference deleted/renamed projects
• _issue_filter_converter that calls Group.objects.get() on user-provided issue short IDs
• resolve_tag_key() called without checking the tag exists in the target dataset
• Passing unvalidated query strings from alert rule subscriptions to Snuba

Safe patterns:

• Validate short IDs before passing to by_qualified_short_id_bulk() -- handle Group.DoesNotExist
• Wrap _issue_filter_converter in try/except and return empty results for invalid filters
• Pre-validate tag existence against dataset schema

Check 4: Value Validation Errors -- 45 issues, 1,000,624 events

ValueError from insufficient input validation: unpacking errors, invalid enum values, missing expected objects, and Pydantic/DRF validation failures.

Red flags:

• SentryAppInstallation.objects.get() in action builders assuming exactly 1 result exists
• AlertRuleWorkflow lookups by ID without handling DoesNotExist
• Tuple unpacking (a, b, c = value.split(":")) on strings that may have fewer separators
• Integer enum lookups without catching ValueError (e.g., DetectorPriorityLevel(value))

Safe patterns:

• Validate expected count before .get(): use .filter() and check .count()
• Wrap tuple unpacking in try/except ValueError or validate length first
• Use try: EnumClass(value) except ValueError: for user-facing enum conversions
• In API endpoints: return 400 for ValueError and validation failures.

Check 5: Type Errors -- 28 issues, 740,106 events

Wrong types passed to functions: iterating over non-iterables, invalid dict keys, None where object expected.

Red flags:

• orjson.dumps(payload) where payload dict may have non-string keys
• list(setting) where setting could be an int (from project.get_option())
• result["key"] = value where result could be None
• Iterating over a value from DB/config that could be int or None instead of list

Safe patterns:

• Type-check before iteration: isinstance(value, (list, str)) before list(value)
• Validate dict keys before JSON serialization: ensure all keys are strings
• None-guard before subscript assignment: if result is not None:
• Defensive option reads with type checking
• In API endpoints: return 400 for type mismatches from user input.

Check 6: Internal API Request Errors -- 71 issues, 829,753 events

ApiError from internal Sentry API calls between services, often caused by stale subscription parameters.

Red flags:

• api.client.get() calls in metric alert chart rendering without handling 400 responses
• Internal API calls that forward stale subscription queries (referencing removed tags/metrics)
• fetch_metric_alert_events_timeseries() in incidents/charts.py without ApiError handling

Safe patterns:

• Wrap internal API calls with try/except ApiError and return graceful fallback
• Validate query parameters before making internal API requests
• Handle 400/500 responses explicitly in chart/visualization code

Check 7: Database Constraint Violations -- 22 issues, 2,962,198 events

IntegrityError and DataError from integer overflow, foreign key violations, and unique constraint violations.

Red flags:

• Incrementing times_seen counter without bounding (integer overflow at ~2.1 billion)
• Deleting MonitorCheckIn records without handling FK constraints from MonitorIncident
• UPDATE on GroupOpenPeriod date ranges where lower bound can exceed upper bound
• save() on models with unique constraints without handling IntegrityError

Safe patterns:

• Cap integer fields before update: min(value, 2_147_483_647) for 32-bit int columns
• Use CASCADE or handle FK constraints before bulk deletion
• Validate date range bounds before saving
• try/except IntegrityError with fallback to get() or update_or_create()

Check 8: Data Parsing & Deserialization -- 19 issues, 272,812 events

JSONDecodeError and ZstdError from parsing external data or stored compressed data.

Red flags:

• json.loads(response.body) without catching JSONDecodeError
• zstd.decompress(data) without handling corrupt frame descriptors
• Parsing VSTS/Azure DevOps webhook bodies that may be truncated
• Assuming API responses are always valid JSON (can be HTML error pages)

Safe patterns:

• Always wrap JSON/compression in try/except with graceful fallback
• Check content-type header before parsing
• Validate response status before parsing body

Check 9: Missing Key Access (KeyError) -- 25 issues, 155,020 events

Accessing dictionary keys or HTTP headers without existence check.

Red flags:

• request.META["HTTP_X_GITLAB_TOKEN"] in webhook handlers (header may be absent)
• request.META["HTTP_X_EVENT_KEY"] in Bitbucket webhook handler
• Dict key lookups with HANDLERS[event_type] without checking the event type is registered
• Tuple unpacking from header values with variable-length splits

Safe patterns:

• request.META.get("HTTP_X_GITLAB_TOKEN") with None check
• HANDLERS.get(event_type) with fallback for unknown event types
• Validate required headers at the top of webhook handler before processing

Check 10: Concurrency and Runtime Bugs -- 23 issues, 38,443 events

Dictionary mutation during iteration, shared mutable state, and unimplemented code paths.

Red flags:

• for key in self._dict: while another thread modifies it (RuntimeError)
• Publishing to shared KafkaPublisher dict that grows unbounded
• dict.pop() or dict[key] = value on a dict being iterated in another thread
• Missing NotImplementedError handlers for new search expression types

Safe patterns:

• dict.copy() before iteration
• Use threading.Lock for shared mutable state
• Implement all code paths before enabling features
• Use list(dict.keys()) for safe iteration when mutation is needed

Check 11: Logic Correctness -- not pattern-based

After checking all known patterns above, reason about the changed code itself:

• Does every code path return the correct type?
• Are all branches of conditionals handled (especially else / default cases)?
• Can any input (None, empty list, 0, empty string) cause unexpected behavior?
• Are there off-by-one errors in loops, slices, or range checks?
• If this code runs concurrently, is shared state protected?

Only report if you can trace a specific input that triggers the bug. Do not report theoretical concerns.

Not a bug — do not flag:

• assert statements enforcing infrastructure invariants — Sentry does not run with python -O, so assertions are always active. Crashing on a violated invariant is intentional.
• Speculative input concerns (e.g., "this URL could be too long", "this header could be malformed") unless you can show the input actually reaches the code path unvalidated. Check for existing validation (host checks, schema validation, DRF serializers) before reporting.

If no checks produced a potential finding, stop and report zero findings. Do not invent issues to fill the report. An empty result is the correct output when the code has no bugs matching these patterns.

Each code location should be reported once under the most specific matching pattern. Do not flag the same line under multiple checks.

Step 3: Report Findings

For each finding, provide the evidence the review harness needs:

• precise location
• severity and confidence
• concrete triggering input or state
• root cause and consequence
• a matching production precedent when available
• a concrete code fix, preferably as a unified diff when the harness supports it

Fix suggestions must include actual code. Never suggest a comment or docstring as a fix.

When suggesting fixes in API endpoints, use appropriate HTTP status codes (404 for not found, 400 for bad input, 409 for conflicts). Never suggest returning 500 intentionally.

Do not prescribe your own output format — the review harness controls the response structure.

Vendored from https://github.com/getsentry/warden-sentry (.agents/skills/sentry-backend-bugs/).

If this skill needs updating, pull changes from that repository.

Concurrency and Runtime Bugs

Table of Contents

• Overview
• Real Examples
• Root Cause Analysis
• Fix Patterns
• Detection Checklist

Overview

23 issues, 38,443 events, 1,320 affected users. Shared mutable state accessed from multiple threads without synchronization, unimplemented code paths hit in production, and list/index access without bounds checking. While lower in issue count, these bugs are persistent -- they fire continuously once triggered and are difficult to reproduce in testing.

Three sub-patterns:

1. Dict mutation during iteration -- One thread iterates a dict while another thread adds/removes keys (RuntimeError, 14 issues) 2. Index out of range -- List access without bounds checking (IndexError, 5 issues) 3. Unimplemented code paths -- NotImplementedError from search expressions or features not yet handled (4 issues)

Real Examples

Example 1: Dictionary changed size during iteration (SENTRY-5BYB) -- unresolved

28,712 events | 0 users

In-app frames:

# sentry/utils/pubsub.py -- publish()
# Called from:
# sentry/monitors/consumers/monitor_consumer.py -- _process_checkin()
# sentry/utils/outcomes.py -- track_outcome()
# RuntimeError: dictionary changed size during iteration

Root cause: A KafkaPublisher instance has an internal dictionary of futures or topics. The publish() method iterates over this dict while another thread concurrently adds new entries. The monitor consumer processes check-ins and calls track_outcome() which publishes to Kafka -- in a multi-threaded consumer, multiple check-ins can be processed concurrently.

Fix:

# Snapshot keys before iteration:
for key in list(self._futures.keys()):
    future = self._futures.get(key)
    if future is not None and future.done():
        self._futures.pop(key, None)

# Or use a lock:
with self._lock:
    done = [k for k, v in self._futures.items() if v.done()]
    for k in done:
        del self._futures[k]

Example 2: IndexError in rate limiting (SENTRY-401M) -- resolved

1,065 events | 353 users

In-app frames:

# sentry/ratelimits/redis.py -- is_limited_with_value()
result = pipeline.execute()
# ...
count = result[idx]  # IndexError: list index out of range

Called from:

# sentry/ratelimits/utils.py -- above_rate_limit_check()
is_limited, current_count = backend.is_limited_with_value(key, limit, window)

Root cause: The Redis pipeline returns fewer results than expected. This can happen if the Redis connection is interrupted mid-pipeline, or if the pipeline commands were partially executed.

Fix:

results = pipeline.execute()
if len(results) <= idx:
    logger.warning("ratelimit.pipeline_incomplete", extra={"expected": idx + 1, "got": len(results)})
    return False, 0  # Default to not rate-limited
count = results[idx]

Actual fix: Resolved -- pipeline results are now validated before access.

Example 3: Process startup timeout (SENTRY-48Q3) -- unresolved

2,773 events | 0 users

In-app frames:

# sentry/spans/consumers/process/flusher.py -- _wait_for_process_to_become_healthy()
if not process.is_alive():
    raise RuntimeError(
        f"process {idx} (shards {shards}) didn't start up in {timeout} seconds"
    )

Root cause: The span processing consumer spawns child processes for sharding. Under load, a child process may take longer than the configured 120-second timeout to start up, causing the parent to raise RuntimeError.

Fix:

# Increase timeout or make it configurable:
timeout = options.get("spans.process.startup_timeout", 300)
# Or implement retry logic:
for attempt in range(max_retries):
    if process.is_alive():
        break
    time.sleep(backoff)
else:
    logger.error("process.startup_timeout", extra={"idx": idx, "shards": shards})
    # Graceful degradation instead of crash

Root Cause Analysis

Fix Patterns

Pattern A: Copy before iteration

# Snapshot the dict keys before iterating:
for key in list(shared_dict.keys()):
    process(shared_dict.get(key))

Pattern B: Use threading.Lock for shared state

import threading

class Publisher:
    def __init__(self):
        self._futures = {}
        self._lock = threading.Lock()

    def publish(self, key, value):
        with self._lock:
            self._futures[key] = produce(value)

    def cleanup(self):
        with self._lock:
            done = [k for k, v in self._futures.items() if v.done()]
            for k in done:
                del self._futures[k]

Pattern C: Bounds checking before index access

# Instead of:
value = results[idx]

# Use:
if idx < len(results):
    value = results[idx]
else:
    logger.warning("unexpected_result_count", extra={"expected": idx + 1, "got": len(results)})
    value = default

Pattern D: Implement all code paths before enabling features

# Instead of:
raise NotImplementedError("Haven't handled all search expressions yet")

# Use:
logger.warning("search.unhandled_expression", extra={"expression": expr})
return default_result  # Graceful fallback

Detection Checklist

Scan the code for these patterns:

• [ ] Any for key in dict: where the dict is accessible from multiple threads -- is it copied first?
• [ ] Any module-level or class-level mutable dicts/lists that are modified at runtime
• [ ] Any dict.pop(), dict[key] = value, or del dict[key] on shared state without a lock
• [ ] Any list index access list[idx] -- is the index bounds-checked?
• [ ] Any pipeline.execute() result access -- is the result list length validated?
• [ ] Any raise NotImplementedError -- is this code reachable in production?
• [ ] Any KafkaPublisher, PubSub, or similar concurrent producer -- is shared state protected?
• [ ] Any child process startup -- is the timeout reasonable under load?

Data Validation and Input Handling

Table of Contents

• Overview
• Real Examples
• Root Cause Analysis
• Fix Patterns
• Detection Checklist

Overview

A broad and impactful category spanning 89 issues across ValueError, KeyError, data parsing, and assertion failures: 1,540,632 events, 10,927 affected users. External input -- from webhook bodies, user parameters, stored data, and binary blobs -- is not validated before use. Includes field length violations, type coercion failures, missing dict keys, corrupt binary data, and violated invariants.

Sub-categories:

1. Value validation (45 issues, 1,000,624 events) -- ValueError from unpacking errors, invalid enum values, missing expected objects 2. Data parsing (19 issues, 272,812 events) -- JSONDecodeError, ZstdError from parsing external or stored data 3. Missing key access (25 issues, 155,020 events) -- KeyError from accessing dict keys or HTTP headers without checking existence 4. Assertion failures (15 issues, 112,495 events) -- AssertionError from bare asserts used as input validation

Real Examples

Example 1: Expected 1 sentry app installation (SENTRY-494A) -- resolved

783,633 events | 0 users

In-app frames:

# sentry/notifications/notification_action/issue_alert_registry/handlers/sentry_app_issue_alert_handler.py
def get_target_identifier(self, action):
    installations = SentryAppInstallation.objects.filter(
        sentry_app__slug=action.target_identifier, ...
    )
    if installations.count() != 1:
        raise ValueError(
            f"Expected 1 sentry app installation for action type: sentry_app, "
            f"target_identifier: {action.target_identifier}"
        )  # CRASHES HERE

Root cause: Alert action builder assumes exactly 1 SentryAppInstallation exists for the target identifier. When the app is uninstalled or there are duplicates, this raises ValueError at massive scale because alert rules fire continuously.

Fix:

installation = SentryAppInstallation.objects.filter(
    sentry_app__slug=action.target_identifier, ...
).first()
if installation is None:
    logger.warning("sentry_app.installation_not_found", ...)
    return None

Actual fix: Resolved -- lookup now handles missing and multiple installations gracefully.

Example 2: Tuple unpacking failure on GitLab webhook (SENTRY-3VCS) -- ignored

66,948 events | 64 users

In-app frames:

# sentry/integrations/gitlab/webhooks.py -- get_gitlab_external_id()
secret, group, _url = token.split(":")  # CRASHES: not enough values to unpack

Root cause: The GitLab token format is expected to be secret:group:url, but some tokens have fewer separators.

Fix:

parts = token.split(":", 2)
if len(parts) != 3:
    raise ValueError(f"Invalid GitLab token format: expected 3 parts, got {len(parts)}")
secret, group, _url = parts

Example 3: KeyError on GitLab/Bitbucket webhook headers (SENTRY-3VCC, SENTRY-3ZXH)

65,805 combined events | 357 users

In-app frames:

# sentry/integrations/gitlab/webhooks.py
return request.META["HTTP_X_GITLAB_TOKEN"]  # CRASHES: KeyError

# sentry/integrations/bitbucket/webhook.py
event = request.META["HTTP_X_EVENT_KEY"]  # CRASHES: KeyError

Root cause: Webhook handlers use direct dict key access on request.META for HTTP headers. External services can send requests without the expected headers.

Fix:

token = request.META.get("HTTP_X_GITLAB_TOKEN")
if token is None:
    return HttpResponse("Missing required header", status=400)

Example 4: ZstdError on attachment decompression (SENTRY-5C5M) -- unresolved

75,658 events | 3,834 users

In-app frames:

# sentry/api/endpoints/event_attachment_details.py -- stream_attachment()
data = zstd.decompress(raw_data)  # CRASHES: Unknown frame descriptor

Root cause: Stored attachment data is corrupted or was stored uncompressed but marked as compressed.

Fix:

try:
    data = zstd.decompress(raw_data)
except zstd.ZstdError:
    logger.warning("attachment.decompress_failed", extra={"attachment_id": attachment.id})
    data = raw_data  # Fall back to raw data

Example 5: JSONDecodeError on VSTS webhook body (SENTRY-5CKF) -- unresolved

30,593 events | 11 users

In-app frames:

# sentry/middleware/integrations/parsers/vsts.py -- get_integration_from_request()
data = json.loads(request.body)  # CRASHES: unexpected end of data

Root cause: Azure DevOps (VSTS) sends webhook bodies that can be truncated. The JSON is valid but incomplete.

Fix:

try:
    data = json.loads(request.body)
except (json.JSONDecodeError, ValueError):
    logger.warning("vsts.webhook.invalid_body", extra={"size": len(request.body)})
    return HttpResponse(status=400)

Example 6: AssertionError in auth login (SENTRY-3VFR) -- resolved

93,882 events | 158 users

In-app frames:

# sentry/web/frontend/auth_login.py -- post()
assert condition  # CRASHES -- assertion used as input validation

Root cause: Bare assert used for validation in the login flow. Assertions can be disabled with python -O and should not be used for input validation in production.

Fix:

if not condition:
    messages.add_message(request, messages.ERROR, "Invalid request")
    return self.redirect(get_login_url())

Actual fix: Resolved -- assertions replaced with explicit validation.

Example 7: Rule must belong to Project assertion (SENTRY-5EMS) -- resolved

8,535 events | 0 users

In-app frames:

# sentry/digests/notifications.py -- build_digest()
for rule in rules:
    assert rule.project_id == project.id, "Rule must belong to Project"  # CRASHES

Root cause: Digest notification builder assumes all rules belong to the same project, but cross-project rule references can occur.

Fix:

for rule in rules:
    if rule.project_id != project.id:
        logger.warning("digest.rule_project_mismatch", extra={...})
        continue

Actual fix: Resolved -- assertion replaced with graceful skip.

Example 8: KeyError on unregistered GitLab event type (SENTRY-3ZWW) -- ignored

45,235 events | 81 users

In-app frames:

# sentry/integrations/gitlab/webhooks.py -- post()
handler = HANDLERS[event_type]  # CRASHES: KeyError: 'Pipeline Hook'

Root cause: GitLab sends webhook events for event types (e.g., "Pipeline Hook") that are not in the HANDLERS dict. The code uses direct key access without checking existence.

Fix:

handler = HANDLERS.get(event_type)
if handler is None:
    logger.info("gitlab.webhook.unhandled_event", extra={"event_type": event_type})
    return HttpResponse(status=204)  # Acknowledge but don't process

Root Cause Analysis

Fix Patterns

Pattern A: Safe header and dict access

# Instead of:
token = request.META["HTTP_X_CUSTOM_HEADER"]
handler = HANDLERS[event_type]

# Use:
token = request.META.get("HTTP_X_CUSTOM_HEADER")
handler = HANDLERS.get(event_type)

Pattern B: Safe tuple unpacking

# Instead of:
a, b, c = value.split(":")

# Use:
parts = value.split(":", 2)
if len(parts) != 3:
    raise ValueError(f"Invalid format: expected 3 parts, got {len(parts)}")
a, b, c = parts

Pattern C: Wrap all deserialization

try:
    data = json.loads(body)
except (json.JSONDecodeError, ValueError):
    return HttpResponse("Invalid JSON", status=400)

Pattern D: Replace assert with explicit validation

# Instead of:
assert rule.project_id == project.id

# Use:
if rule.project_id != project.id:
    logger.warning("rule.project_mismatch", extra={"rule_id": rule.id})
    continue

Pattern E: Validate binary data

try:
    data = zstd.decompress(raw_data)
except zstd.ZstdError:
    data = raw_data  # Fall back

Pattern F: Safe enum conversion

try:
    level = DetectorPriorityLevel(value)
except ValueError:
    level = DetectorPriorityLevel.DEFAULT

Detection Checklist

Scan the code for these patterns:

• [ ] Any request.META["HTTP_X_..."] -- use .get() instead
• [ ] Any dict[key] lookup on handler registries or maps -- use .get() with fallback
• [ ] Any a, b, c = value.split(...) -- validate part count first
• [ ] Any bare assert statement -- replace with explicit validation and error handling
• [ ] Any json.loads() or orjson.loads() -- wrapped in try/except?
• [ ] Any zstd.decompress() or decompression -- handles corrupt data?
• [ ] Any Model.objects.filter(...).count() != 1 followed by a raise -- handle 0 and >1 gracefully?
• [ ] Any int(), float(), EnumClass() on user input -- wrapped in try/except?
• [ ] Any webhook handler that accesses request.body -- handles empty/truncated bodies?
• [ ] Any get_or_create() -- are field values validated against max_length first?

Database Integrity Violations

Table of Contents

• Overview
• Real Examples
• Root Cause Analysis
• Fix Patterns
• Detection Checklist

Overview

31 issues (22 constraint violations + 9 duplicate object collisions), 2,972,784 events, 940 affected users. Race conditions between concurrent requests cause duplicate key violations, foreign key violations during cleanup, integer overflow on counter fields, and MultipleObjectsReturned from get() calls where uniqueness assumptions are violated.

Three main sub-patterns:

1. DataError from integer overflow -- Counter fields like times_seen overflow the 32-bit integer range (~2.1 billion) 2. IntegrityError from FK violations -- Bulk deletion of parent records while child records still reference them 3. MultipleObjectsReturned -- get() assumes exactly one match, but data has duplicates

Real Examples

Example 1: Integer out of range on times_seen counter (SENTRY-4E5F) -- resolved

1,753,743 events | 0 users

In-app frames:

# sentry/db/models/query.py -- update()
# SQL: UPDATE "sentry_groupedmessage" SET "times_seen" = ("sentry_groupedmessage"."times_seen" + 1)
# DataError: integer out of range

Called from:

# sentry/buffer/base.py -- process()
Group.objects.filter(id=group_id).update(times_seen=F("times_seen") + count)

Root cause: The times_seen field on Group is a standard 32-bit integer. For very active groups, the counter exceeds 2,147,483,647 and the Postgres UPDATE fails with DataError: integer out of range.

Fix:

# Cap the increment to prevent overflow
from django.db.models import F, Value
from django.db.models.functions import Least

Group.objects.filter(id=group_id).update(
    times_seen=Least(F("times_seen") + count, Value(2_147_483_647))
)

Actual fix: Resolved -- either the field was migrated to BigInteger or the increment is now capped.

Example 2: FK violation during MonitorCheckIn cleanup (SENTRY-5DCR) -- resolved

187,518 events | 0 users

In-app frames:

# sentry/utils/query.py -- bulk_delete_objects()
# IntegrityError: update or delete on table "sentry_monitorcheckin" violates
# foreign key constraint "sentry_monitorincide_..." on table "sentry_monitorincident"

Called from:

# sentry/runner/commands/cleanup.py -- multiprocess_worker()
bulk_delete_objects(MonitorCheckIn, ...)

Root cause: The cleanup task bulk-deletes old MonitorCheckIn records, but MonitorIncident records still reference them via a foreign key. The deletion does not check for or cascade child references.

Fix:

# Delete child references first, or use CASCADE
MonitorIncident.objects.filter(
    checkin_id__in=checkin_ids_to_delete
).update(checkin_id=None)  # Or delete incidents first
bulk_delete_objects(MonitorCheckIn, ...)

Actual fix: Resolved -- cleanup now handles FK constraints before deletion.

Example 3: Date range lower bound exceeds upper bound (SENTRY-4EDG) -- ignored

753,527 events | 0 users

In-app frames:

# sentry/models/groupopenperiod.py -- close_open_period()
# DataError: range lower bound must be less than or equal to range upper bound
# SQL: UPDATE "sentry_groupopenperiod" SET ...

Root cause: The GroupOpenPeriod model uses a date range field. When closing an open period, the end timestamp can be earlier than the start timestamp due to clock skew or race conditions in the event pipeline.

Fix:

def close_open_period(self, end_time):
    if end_time < self.start_time:
        end_time = self.start_time  # Clamp to valid range
    self.date_range = DateTimeTZRange(self.start_time, end_time)
    self.save()

Example 4: MultipleObjectsReturned on Repository (SENTRY-3W17) -- unresolved

2,391 events | 3 users

In-app frames:

# sentry_plugins/heroku/plugin.py -- set_refs()
repo = Repository.objects.get(
    name=repo_name, organization_id=org_id
)  # Repository.MultipleObjectsReturned!

Root cause: The (name, organization_id) combination is not unique at the database level (or became non-unique through a migration gap). The code uses get() which raises when more than one match exists.

Fix:

repo = Repository.objects.filter(
    name=repo_name, organization_id=org_id,
).order_by("-date_added").first()
if repo is None:
    raise Repository.DoesNotExist()

Example 5: SentryAppInstallation.MultipleObjectsReturned (SENTRY-5HSD) -- resolved

2,554 events | 305 users

In-app frames:

# sentry/sentry_apps/services/app/impl.py -- find_installation_by_proxy_user()
installation = SentryAppInstallation.objects.get(
    sentry_app=sentry_app, ...
)  # MultipleObjectsReturned: get() returned more than one -- it returned 4!

Root cause: A SentryApp can have multiple installations (e.g., installed, uninstalled, re-installed) and the query does not filter by status.

Fix:

installation = SentryAppInstallation.objects.filter(
    sentry_app=sentry_app,
    status=SentryAppInstallationStatus.INSTALLED,
    ...
).first()

Actual fix: Resolved -- query now filters by status and uses .first().

Example 6: ExternalActor.MultipleObjectsReturned in notifications (SENTRY-43YW) -- resolved

2,250 events | 0 users

In-app frames:

# sentry/integrations/notifications.py -- _get_channel_and_integration_by_team()
actor = ExternalActor.objects.get(
    team_id=team_id, integration_id=integration_id, ...
)  # MultipleObjectsReturned!

Root cause: An ExternalActor (Slack channel mapping for a team) was duplicated, likely through a data migration or re-linking.

Fix:

actor = ExternalActor.objects.filter(
    team_id=team_id, integration_id=integration_id, ...
).first()

Actual fix: Resolved -- uses .first() instead of .get().

Root Cause Analysis

Fix Patterns

Pattern A: Cap integer fields before update

from django.db.models import F, Value
from django.db.models.functions import Least

Model.objects.filter(id=pk).update(
    counter=Least(F("counter") + increment, Value(2_147_483_647))
)

Pattern B: Handle FK constraints in bulk deletion

# Delete or nullify child references first
ChildModel.objects.filter(parent_id__in=ids_to_delete).delete()
# Then delete parents
ParentModel.objects.filter(id__in=ids_to_delete).delete()

Pattern C: Validate range bounds

if end_time < start_time:
    end_time = start_time  # or raise ValueError

Pattern D: filter().first() instead of get()

# Instead of:
obj = Model.objects.get(name=name, org=org)

# Use:
obj = Model.objects.filter(name=name, org=org).order_by("-date_added").first()
if obj is None:
    raise Model.DoesNotExist()

Pattern E: Insert-or-fetch for concurrent inserts

from django.db import IntegrityError

try:
    obj = Model.objects.create(**fields)
except IntegrityError:
    obj = Model.objects.get(**unique_fields)

Detection Checklist

Scan the code for these patterns:

• [ ] Any F("field") + increment on integer fields -- can the field overflow 2^31?
• [ ] Any bulk deletion (bulk_delete_objects, queryset.delete()) -- are there FK constraints on child tables?
• [ ] Any date range field updates -- can lower bound exceed upper bound?
• [ ] Any .get() call -- is MultipleObjectsReturned possible? Check if the filter fields are actually unique at DB level
• [ ] Any .save() on a model with unique constraints -- is IntegrityError handled?
• [ ] Any get_or_create() in concurrent context -- wrapped in try/except IntegrityError?
• [ ] Any consumer/worker code processing messages -- can the same message be processed concurrently?

Integration and Webhook Handling Errors

Table of Contents

• Overview
• Real Examples
• Root Cause Analysis
• Fix Patterns
• Detection Checklist

Overview

96 issues spanning SentryApp webhook errors (25 issues, 6.6M events) and API request errors (71 issues, 830K events), totaling 7,459,209 events, 8,724 affected users. Integration webhooks, SentryApp callbacks, internal API requests, and external API interactions fail on unexpected payloads, missing state, or stale configuration. This is the highest-volume cluster because a single broken integration can generate millions of events from continuous alert rule firing.

Key sub-patterns:

1. Missing service hook or installation -- SentryApp uninstalled but webhooks still fire or alert rules still reference it 2. Event not eligible for webhook -- Webhook delivery attempted for events not in the service hook's event list 3. Internal API errors from stale parameters -- Internal API calls forwarding stale subscription queries that reference removed tags/metrics 4. JSON decode on empty or truncated body -- Integration partner sends empty body, HTML error page, or truncated JSON

Real Examples

Example 1: SentryAppSentryError event_not_in_servicehook (SENTRY-414F) -- resolved

5,419,218 events | 0 users

In-app frames:

# sentry/sentry_apps/tasks/sentry_apps.py -- send_webhooks()
# Line 796: SentryAppSentryError: event_not_in_servicehook

Root cause: Workflow notification tasks attempt to send webhooks to SentryApps for event types that are not in the service hook's configured event list. The task checks the event type against the service hook's events but raises an error instead of silently skipping. At 5.4M events this is the single highest-volume code bug.

Fix:

def send_webhooks(sentry_app, event, ...):
    servicehooks = ServiceHook.objects.filter(
        application_id=sentry_app.application_id,
    )
    for hook in servicehooks:
        if event_type not in hook.events:
            continue  # Skip -- this hook doesn't subscribe to this event type
        _deliver_webhook(hook, event, ...)

Actual fix: Resolved -- event eligibility check now skips instead of raising.

Example 2: SentryAppSentryError missing_servicehook (SENTRY-41EN) -- resolved

391,256 events | 0 users

In-app frames:

# sentry/sentry_apps/tasks/sentry_apps.py -- send_webhooks()
# Line 785: SentryAppSentryError: missing_servicehook

Root cause: A SentryApp was uninstalled (removing the ServiceHook), but existing alert rules still trigger webhook delivery tasks. The task cannot find the service hook and raises.

Fix:

servicehook = ServiceHook.objects.filter(
    application_id=sentry_app.application_id,
    actor_id=sentry_app.proxy_user_id,
).first()
if servicehook is None:
    logger.info("sentry_app.webhook.missing_servicehook", extra={"sentry_app_id": sentry_app.id})
    return  # App was uninstalled, skip delivery

Actual fix: Resolved -- missing hook now results in a skip rather than an error.

Example 3: Internal API error from stale metric subscription (SENTRY-55BH) -- ignored

340,371 events | 0 users

In-app frames:

# sentry/incidents/charts.py -- fetch_metric_alert_events_timeseries()
response = client.get(url, params=params)  # ApiError: status=400
# body={'detail': ErrorDetail(string='transaction.duration is not a tag in the metrics dataset')}

Called from metric alert action triggers:

# sentry/workflow_engine/tasks -- trigger_action()
chart_data = fetch_metric_alert_events_timeseries(...)  # Crashes

Root cause: A metric alert subscription references transaction.duration which is not a valid tag in the metrics dataset (it is a string type, not numeric). When the alert fires, the action tries to render a chart by querying the internal API with the same stale parameters. The internal API returns a 400 error that is not caught.

Fix:

try:
    chart_data = fetch_metric_alert_events_timeseries(
        subscription_query=subscription.query, ...
    )
except ApiError as e:
    logger.warning(
        "incidents.charts.fetch_failed",
        extra={"subscription_id": subscription.id, "error": str(e)},
    )
    chart_data = None  # Proceed without chart

Example 4: Internal API error from apdex threshold incompatibility (SENTRY-54VM) -- resolved

65,927 events | 0 users

Same pattern as above but for apdex queries:

# sentry/incidents/charts.py -- fetch_metric_alert_events_timeseries()
# ApiError: status=400 body={'detail': 'Cannot query apdex with a threshold parameter on the metrics dataset'}

Root cause: Old alert rules created with the events dataset reference apdex() with threshold parameters. When these rules fire and the chart render uses the metrics dataset, the query is incompatible.

Actual fix: Resolved -- chart rendering now handles API errors gracefully.

Root Cause Analysis

Fix Patterns

Pattern A: Check event eligibility before webhook delivery

def send_webhooks(sentry_app, event_type, ...):
    hooks = ServiceHook.objects.filter(application_id=sentry_app.application_id)
    for hook in hooks:
        if event_type not in hook.events:
            continue  # Not subscribed to this event type
        _deliver(hook, ...)

Pattern B: Handle missing installations gracefully

hook = ServiceHook.objects.filter(
    application_id=app.application_id,
    actor_id=app.proxy_user_id,
).first()
if hook is None:
    return  # App uninstalled

Pattern C: Catch internal API errors in action triggers

try:
    chart_data = fetch_metric_alert_events_timeseries(...)
except ApiError:
    chart_data = None  # Send alert without chart attachment

Pattern D: Validate subscription query compatibility before use

def validate_subscription_query(subscription):
    """Check if the subscription query is compatible with the current dataset."""
    try:
        build_query(subscription.query, dataset=subscription.dataset)
    except (IncompatibleMetricsQuery, InvalidSearchQuery):
        subscription.mark_invalid()
        return False
    return True

Pattern E: Safe JSON parsing for webhooks

def parse_webhook_body(request):
    if not request.body:
        return {}
    try:
        return orjson.loads(request.body)
    except orjson.JSONDecodeError:
        logger.warning("webhook.invalid_json", extra={"path": request.path})
        return None

Detection Checklist

Scan the code for these patterns:

• [ ] Any SentryApp webhook delivery -- does it check event eligibility against the service hook's event list?
• [ ] Any ServiceHook or SentryAppInstallation lookup in webhook tasks -- is DoesNotExist handled?
• [ ] Any internal API call in metric alert action triggers -- is ApiError caught?
• [ ] Any fetch_metric_alert_events_timeseries() call -- does it handle 400 responses?
• [ ] Any json.loads() on request.body in webhook handlers -- is JSONDecodeError caught?
• [ ] Any external API response parsed as JSON -- is the status code and content-type checked?
• [ ] Any subscription query forwarded to internal API -- has compatibility been validated?
• [ ] Any alert rule referencing a SentryApp -- what happens when the app is uninstalled?

Missing Records and Stale References

Table of Contents

• Overview
• Real Examples
• Root Cause Analysis
• Fix Patterns
• Detection Checklist

Overview

The most impactful code-level bug category in the Sentry backend: 81 issues, 1,403,592 events, 10,727 affected users. The pattern is consistent -- code calls .get() on a Django model assuming the record exists, but it has been deleted, merged, or never created.

The most common sources of stale IDs:

1. Snuba/ClickHouse query results -- Snuba stores issue IDs, project IDs, and group IDs that may be deleted from Postgres before Snuba data expires 2. Workflow engine references -- Detectors, subscriptions, and alert rules reference objects deleted asynchronously 3. Integration state -- SentryAppInstallation, ServiceHook, or ExternalActor deleted while alert rules still reference them 4. Cross-silo references -- Cell silo holds IDs that reference control silo objects (or vice versa) that may be deleted asynchronously 5. Cached foreign keys -- A ProjectKey cached in Redis still references a project_id for a deleted project 6. Monitor/cron references -- Environment objects referenced by monitors that may be deleted

Real Examples

Example 1: Detector.DoesNotExist in workflow engine (SENTRY-5D9J) -- resolved

610,142 events | 0 users

In-app frames:

# sentry/workflow_engine/processors/detector.py -- _get_detector_for_event()
def _get_detector_for_event(event_data):
    detector_id = event_data.get("detector_id")
    try:
        return Detector.objects.get(id=detector_id)  # CRASHES HERE
    except Detector.DoesNotExist:
        raise  # Re-raises without handling

Root cause: Workflow events reference detector IDs that have been deleted. The process_workflows_event task receives events from a queue with detector IDs, but detectors can be deleted between event creation and processing.

Fix pattern:

detector = Detector.objects.filter(id=detector_id).first()
if detector is None:
    logger.warning("detector.not_found", extra={"detector_id": detector_id})
    return  # Skip processing for deleted detectors

Actual fix: Resolved -- detector lookup now handles the DoesNotExist case gracefully.

Example 2: Environment.DoesNotExist in monitor consumer (SENTRY-3VDX) -- resolved

146,432 events | 0 users

In-app frames:

# sentry/monitors/models.py -- get_environment()
def get_environment(self):
    return Environment.objects.get(id=self.environment_id)  # CRASHES HERE

Called from:

# sentry/monitors/logic/incident_occurrence.py -- send_incident_occurrence()
environment = monitor.get_environment()

Root cause: The monitor checkin references an environment that has been deleted. The Environment.objects.get() call has no DoesNotExist handler.

Fix pattern:

def get_environment(self):
    try:
        return Environment.objects.get(id=self.environment_id)
    except Environment.DoesNotExist:
        return None

Actual fix: Resolved -- environment lookup now returns None for deleted environments.

Example 3: Subscription.DoesNotExist in billing tasks (SENTRY-4DEQ) -- resolved

72,700 events | 0 users

In-app frames:

# getsentry/billing/tasks/usagebuffer.py -- flush_usage_buffer()
subscription = Subscription.objects.get(id=subscription_id)  # CRASHES HERE

Root cause: Billing usage buffer tasks reference subscription IDs that have been cancelled/deleted between task scheduling and execution.

Fix pattern:

try:
    subscription = Subscription.objects.get(id=subscription_id)
except Subscription.DoesNotExist:
    logger.info("subscription.not_found", extra={"subscription_id": subscription_id})
    return  # Nothing to flush for a deleted subscription

Actual fix: Resolved -- task now handles missing subscriptions gracefully.

Root Cause Analysis

Fix Patterns

Pattern A: Graceful skip for async task processing

When a celery task or consumer processes an event referencing an object by ID, handle the case where the object was deleted between event creation and processing.

def process_workflow_event(event_data):
    detector = Detector.objects.filter(id=event_data["detector_id"]).first()
    if detector is None:
        logger.info("detector.deleted", extra={"detector_id": event_data["detector_id"]})
        return
    # proceed with detector

Pattern B: Filter query instead of get

When you need a single object that might not exist, use .filter().first() instead of .get().

# Instead of:
project = Project.objects.get(id=key.project_id)

# Use:
project = Project.objects.filter(id=key.project_id).first()
if project is None:
    return handle_missing_project()

Pattern C: Batch lookups with graceful skip

When processing a list of items, prefetch and skip missing records instead of crashing the entire batch.

# Instead of:
for event in events:
    group = Group.objects.get(id=event["issue.id"])  # Crashes on missing

# Use:
group_ids = [e["issue.id"] for e in events]
groups = {g.id: g for g in Group.objects.filter(id__in=group_ids)}
for event in events:
    group = groups.get(event["issue.id"])
    if group is None:
        continue
    process(group)

Pattern D: Cascade cleanup on deletion

When deleting a parent object, ensure downstream references are cleaned up.

# When deleting an environment:
Environment.objects.filter(id=env_id).delete()
# Also update monitors that reference this environment
Monitor.objects.filter(environment_id=env_id).update(environment_id=None)

Detection Checklist

Scan the code for these patterns:

• [ ] Any .get() call on a Django model manager -- does it have a DoesNotExist handler?
• [ ] Any .get_from_cache() call -- does it handle the case where the cached FK target is deleted?
• [ ] Any code that uses IDs from Snuba, Redis, Kafka, or task queues to look up Postgres records
• [ ] Any code in workflow engine that looks up Detectors, AlertRuleWorkflows, or Subscriptions by ID
• [ ] Any code in monitor/cron consumers that looks up Environments or MonitorCheckIns
• [ ] Any code in billing tasks that looks up Subscriptions by ID
• [ ] Chained lookups: first .get() succeeds, second .get() on a related object fails
• [ ] Batch serialization code that calls .get() in a loop without try/except

Null Reference and Type Errors

Table of Contents

• Overview
• Real Examples
• Root Cause Analysis
• Fix Patterns
• Detection Checklist

Overview

A high-impact bug category spanning 43 issues across TypeError and AttributeError, 767,662 events, 2,919 affected users. Code assumes a value has a specific type or is non-None, but runtime data violates that assumption. These are particularly insidious because they often only trigger for specific data shapes in production.

Common shapes:

1. None where dict expected -- A serializer or function returns None instead of a dict, then caller does result["key"] = value 2. Int where iterable expected -- A project option or config value is stored as an int but code calls list(value) or iterates over it 3. Missing attribute on request -- Code accesses request.auth on a Django WSGIRequest that has not gone through DRF authentication 4. Non-string dict keys -- Payload dicts with integer keys passed to JSON serializers that require string keys 5. Wrong return type from option/config -- project.get_option() returns a different type than expected 6. Str where object expected -- Code receives a string ID where it expects a model instance with .id attribute

Real Examples

Example 1: Dict key must be str in data forwarding (SENTRY-5HY1) -- resolved

596,437 events | 0 users

In-app frames:

# sentry/integrations/data_forwarding/amazon_sqs/forwarder.py -- forward_event()
s3_put_object(
    Bucket=s3_bucket,
    Body=orjson.dumps(payload, option=orjson.OPT_UTC_Z).decode(),  # CRASHES HERE
    Key=key,
)

Root cause: The event payload dict contains non-string keys (likely integer keys from event data). orjson.dumps() requires all dict keys to be strings, unlike json.dumps() which auto-converts them.

Fix:

# Ensure all keys are strings before serialization
def _stringify_keys(obj):
    if isinstance(obj, dict):
        return {str(k): _stringify_keys(v) for k, v in obj.items()}
    if isinstance(obj, list):
        return [_stringify_keys(item) for item in obj]
    return obj

payload = _stringify_keys(payload)
Body = orjson.dumps(payload, option=orjson.OPT_UTC_Z).decode()

Actual fix: Resolved -- payload keys are now converted to strings before serialization.

Example 2: Int object is not iterable in filter config (SENTRY-5JS8) -- unresolved

60,654 events | 0 users

In-app frames:

# sentry/relay/config/__init__.py -- _filter_option_to_config_setting()
if setting == "1":
    ret_val["options"] = ["default"]
else:
    # new style filter, per legacy browser type handling
    ret_val["options"] = list(setting)   # CRASHES HERE when setting is an int

Root cause: project.get_option("filters:legacy-browsers") returned an integer value instead of a string. The code calls list(setting) which works for strings (produces a list of characters) but throws TypeError for ints. This is a data shape assumption -- the option was stored as an int by an older code path.

Fix:

if setting == "1" or setting == 1:
    ret_val["options"] = ["default"]
elif isinstance(setting, str):
    ret_val["options"] = list(setting)
elif isinstance(setting, (list, tuple)):
    ret_val["options"] = list(setting)
else:
    ret_val["options"] = ["default"]

Example 3: NoneType item assignment in event response (SENTRY-3Z3P) -- unresolved

32,246 events | 158 users

In-app frames:

# sentry/issues/endpoints/project_event_details.py -- wrap_event_response()
event_data["nextEventID"] = next_event_id      # CRASHES HERE
event_data["previousEventID"] = prev_event_id
return event_data

Root cause: event_data is None. The event serializer returned None instead of a dict (likely because the event data was missing or corrupt), but wrap_event_response assumes it always gets a dict back.

Fix:

event_data = serialize_event(event, ...)
if event_data is None:
    raise NotFound("Event data could not be serialized")

event_data["nextEventID"] = next_event_id
event_data["previousEventID"] = prev_event_id
return event_data

Example 4: WSGIRequest has no attribute 'auth' (SENTRY-3VXH) -- unresolved

18,241 events | 47 users

In-app frames:

# sentry/middleware/__init__.py -- is_frontend_request()
return bool(request.COOKIES) and request.auth is None   # CRASHES HERE

Root cause: is_frontend_request() is called from middleware that runs on all requests, including Django views that do not go through DRF authentication. Plain WSGIRequest objects do not have an auth attribute.

Fix:

def is_frontend_request(request):
    return bool(request.COOKIES) and getattr(request, 'auth', None) is None

Example 5: 'str' object has no attribute 'id' in release search (SENTRY-548A / SENTRY-3Z3X)

5,250 combined events | 283 users (SENTRY-3Z3X resolved, SENTRY-548A unresolved)

In-app frames:

# sentry/search/utils.py -- _run_latest_release_query()
releases = Release.objects.filter(...)
return [r.id for r in releases]  # Works

# But later in get_latest_release():
return [r.id for r in results]  # CRASHES when results contains strings

Root cause: The _run_latest_release_query function can return a list of strings (release version strings) instead of Release objects in certain code paths. Callers then access .id on strings.

Fix:

# Ensure consistent return type
results = _run_latest_release_query(...)
if results and isinstance(results[0], str):
    # Convert version strings to Release objects
    releases = Release.objects.filter(version__in=results, ...)
    return [r.id for r in releases]

Root Cause Analysis

Fix Patterns

Pattern A: Guard before subscript assignment

# Instead of:
data["key"] = value

# Use:
if data is not None:
    data["key"] = value
# Or raise early:
if data is None:
    raise ValueError("Expected dict, got None")

Pattern B: Type-check before iteration

# Instead of:
items = list(value)

# Use:
if isinstance(value, str):
    items = list(value)
elif isinstance(value, (list, tuple)):
    items = list(value)
elif isinstance(value, int):
    items = [str(value)]
else:
    items = []

Pattern C: getattr for optional request attributes

# Instead of:
request.auth

# Use:
getattr(request, 'auth', None)

Pattern D: Ensure consistent return types

# If a function can return different types, normalize at the boundary:
def get_items(query):
    results = _internal_query(query)
    if not results:
        return []
    # Ensure we always return model instances, not strings
    if isinstance(results[0], str):
        return Model.objects.filter(name__in=results)
    return results

Pattern E: Stringify dict keys before JSON serialization

# When using orjson (strict about key types):
def safe_serialize(data):
    if isinstance(data, dict):
        return {str(k): safe_serialize(v) for k, v in data.items()}
    if isinstance(data, list):
        return [safe_serialize(item) for item in data]
    return data

Detection Checklist

Scan the code for these patterns:

• [ ] Any result["key"] = ... -- can result be None?
• [ ] Any list(value) or for x in value: -- can value be an int, None, or unexpected type?
• [ ] Any request.auth, request.user, or other DRF-specific attributes -- is this code reachable from non-DRF views?
• [ ] Any project.get_option() or organization.get_option() return value used without type checking
• [ ] Any function that returns different types (dict or None, str or object) -- do all callers handle both?
• [ ] Any orjson.dumps() call -- are all dict keys guaranteed to be strings?
• [ ] Middleware or utility functions called on all request paths -- do they assume DRF request attributes?
• [ ] Any .id attribute access on a variable that could be a string instead of a model instance

Query and Subscription Validation Errors

Table of Contents

• Overview
• Real Examples
• Root Cause Analysis
• Fix Patterns
• Detection Checklist

Overview

The highest-event-count cluster combining metric subscription errors and search query validation: 170 issues, 5,036,970 events, 2,598 affected users. Snuba subscriptions and search queries reference tags, fields, or functions that are invalid for the target dataset. These fire continuously -- a single bad subscription generates thousands of events per hour.

Two main sub-patterns:

1. Metric subscription query errors (113 issues, 3,035,640 events) -- SubscriptionError when creating or updating alert subscriptions in Snuba with incompatible query parameters 2. Search query validation errors (57 issues, 2,001,330 events) -- InvalidSearchQuery from user-provided or subscription-stored query filters referencing invalid values, deleted issues, or unresolved tags

Real Examples

Example 1: SubscriptionError -- tag not in metrics dataset (SENTRY-413P) -- resolved

531,826 events | 0 users

In-app frames:

# sentry/search/events/builder/metrics.py -- resolve_tag_key()
# IncompatibleMetricsQuery: customerType is not a tag in the metrics dataset

# Wrapped and re-raised as:
# sentry/snuba/tasks.py -- _create_in_snuba()
# SubscriptionError: customerType is not a tag in the metrics dataset

Root cause: A metric alert subscription uses customerType as a filter tag, but this tag does not exist in the metrics dataset. The subscription was created when the events dataset was the default, which allowed arbitrary tag names. After migration to the metrics dataset, only known metric tags are valid.

Fix:

def _create_in_snuba(subscription):
    try:
        snuba_query = build_snuba_query(subscription)
        result = _snuba_pool.submit(snuba_query)
    except (IncompatibleMetricsQuery, InvalidSearchQuery) as e:
        logger.warning(
            "subscription.incompatible_query",
            extra={"subscription_id": subscription.id, "error": str(e)},
        )
        subscription.update(status=QuerySubscription.Status.DISABLED.value)
        return  # Disable instead of crash

Actual fix: Resolved -- subscription creation now handles incompatible queries.

Example 2: SubscriptionError -- invalid function parameter type (SENTRY-4DAA) -- resolved

294,145 events | 0 users

In-app frames:

# sentry/search/eap/resolver.py -- resolve_function()
# InvalidSearchQuery: transaction.duration is invalid for parameter 1 in p95.
# Its a string type field, but it must be one of: ...

# Re-raised as:
# sentry/snuba/tasks.py -- _create_in_snuba()
# SubscriptionError: transaction.duration is invalid for parameter 1 in p95.

Root cause: An alert subscription uses p95(transaction.duration) on the metrics dataset. In the metrics dataset, transaction.duration is stored as a string (tag), not a numeric field, making it incompatible with aggregate functions like p95().

Fix: Same as Example 1 -- validate function parameter types against the dataset schema before subscription creation.

Actual fix: Resolved -- subscription creation validates field types.

Example 3: SubscriptionError -- apdex threshold incompatibility (SENTRY-413N) -- resolved

276,062 events | 0 users

In-app frames:

# sentry/search/events/datasets/metrics.py -- _resolve_apdex_function()
# IncompatibleMetricsQuery: Cannot query apdex with a threshold parameter on the metrics dataset

# Re-raised as:
# sentry/snuba/tasks.py -- _create_in_snuba()
# SubscriptionError: Cannot query apdex with a threshold parameter on the metrics dataset

Root cause: Old alert rules use apdex() with a threshold parameter (e.g., apdex(300)). The metrics dataset does not support per-query thresholds for apdex -- it uses the project-level threshold instead.

Actual fix: Resolved -- apdex function resolution now handles threshold incompatibility.

Example 4: InvalidSearchQuery -- deleted issue short ID in subscription (SENTRY-3TYF) -- resolved

1,221,510 events | 0 users

In-app frames:

# sentry/search/events/datasets/discover.py -- _issue_filter_converter()
def _issue_filter_converter(search_filter, ...):
    # Resolves short IDs like 'PROJ-ABC' to Group IDs
    groups = Group.objects.by_qualified_short_id_bulk(...)
    # Group.DoesNotExist raised when short ID references a deleted project

# Re-raised as:
# InvalidSearchQuery: Invalid value '['PROJ-ABC']' for 'issue:' filter

Called from the subscription consumer:

# sentry/incidents/utils/process_update_helpers.py -- get_aggregation_value()

Root cause: Metric alert subscriptions stored query strings containing issue short IDs (e.g., issue:PROJ-ABC). When the referenced project or issue is deleted, the _issue_filter_converter tries to resolve the short ID via by_qualified_short_id_bulk(), which calls Group.objects.get() and raises DoesNotExist. This is wrapped as InvalidSearchQuery.

This is the single highest-event resolved issue in the search query cluster at 1.2M events -- the subscription fires continuously because the issue is permanently deleted and the query can never succeed.

Fix:

def _issue_filter_converter(search_filter, ...):
    try:
        groups = Group.objects.by_qualified_short_id_bulk(org_id, short_ids)
    except Group.DoesNotExist:
        # Short ID references a deleted project/issue
        return Condition(Column("group_id"), Op.IN, [])  # Empty result set

Actual fix: Resolved -- issue filter converter now handles deleted short IDs gracefully.

Root Cause Analysis

Fix Patterns

Pattern A: Validate subscription queries before creation

def create_subscription(query, dataset):
    try:
        # Dry-run the query to validate it
        build_snuba_query(query, dataset=dataset)
    except (IncompatibleMetricsQuery, InvalidSearchQuery) as e:
        raise ValidationError(f"Invalid subscription query: {e}")

Pattern B: Handle incompatible queries at subscription processing time

def process_subscription_update(subscription, update):
    try:
        result = execute_query(subscription.query, dataset=subscription.dataset)
    except (IncompatibleMetricsQuery, InvalidSearchQuery):
        subscription.update(status=QuerySubscription.Status.DISABLED)
        logger.warning("subscription.disabled_incompatible_query", ...)
        return

Pattern C: Graceful DoesNotExist in filter converters

def _issue_filter_converter(search_filter, ...):
    try:
        groups = Group.objects.by_qualified_short_id_bulk(org_id, short_ids)
    except Group.DoesNotExist:
        return Condition(Column("group_id"), Op.IN, [])  # Empty result

Pattern D: Validate tag existence before query

def resolve_tag_key(tag_name, dataset):
    if tag_name not in get_valid_tags(dataset):
        raise InvalidSearchQuery(
            f"'{tag_name}' is not a valid tag for the {dataset} dataset"
        )

Detection Checklist

Scan the code for these patterns:

• [ ] Any _create_in_snuba or subscription creation -- does it validate query compatibility with the target dataset?
• [ ] Any resolve_tag_key() call -- does it handle tags that don't exist in the dataset?
• [ ] Any resolve_function() or resolve_snql_function() -- does it validate parameter types?
• [ ] Any _issue_filter_converter or by_qualified_short_id_bulk() -- does it handle deleted groups?
• [ ] Any alert subscription that stores a raw query string -- is the query re-validated on use?
• [ ] Any build_snuba_query() for subscription processing -- does it catch IncompatibleMetricsQuery?
• [ ] Any code that passes user-provided filters to Snuba -- are the filter values validated?
• [ ] Any subscription update task -- what happens when the subscription query is no longer valid?

URL Safety and Routing Errors

Table of Contents

• Overview
• Real Examples
• Root Cause Analysis
• Fix Patterns
• Detection Checklist

Overview

15 issues, 126,318 events, 3,602 affected users. Redirect URLs that exceed safety limits, missing URL schemes in external URLs, and routing mismatches from URL construction.

Three sub-patterns:

1. DisallowedRedirect (8 issues, 98,940 events) -- Redirect URLs exceeding the 2048-character safety limit, all resolved 2. MissingSchema (5 issues, 25,548 events) -- External URLs stored without a scheme (https://), causing requests library to fail 3. NoReverseMatch (2 issues, 1,830 events) -- Django URL routing failures from invalid parameters

Real Examples

Example 1: Unsafe redirect exceeding 2048 characters (SENTRY-5D1A) -- resolved

49,799 events | 897 users

In-app frames:

# sentry/middleware/access_log.py -- middleware()
# -> sentry/middleware/subdomain.py -- __call__()
# DisallowedRedirect: Unsafe redirect exceeding 2048 characters

Root cause: Users access Sentry URLs with very long query strings or path segments (often from malformed links, bots, or security scanners). The middleware chain attempts to redirect these requests (e.g., subdomain redirect, customer domain redirect, marketing landing redirect) but the resulting redirect URL exceeds the 2048-character safety limit. The DisallowedRedirect exception is thrown by Sentry's redirect safety middleware.

Fix:

def safe_redirect(url, max_length=2048):
    if len(url) > max_length:
        # Truncate to base URL without query string
        parsed = urlparse(url)
        base_url = f"{parsed.scheme}://{parsed.netloc}{parsed.path}"
        if len(base_url) > max_length:
            return HttpResponseBadRequest("URL too long")
        return redirect(base_url)
    return redirect(url)

Actual fix: Resolved -- all 8 DisallowedRedirect issues were fixed. The redirect paths now handle overly long URLs gracefully.

Example 2: Unsafe redirect in customer domain middleware (SENTRY-5D1G) -- resolved

29,386 events | 81 users

In-app frames:

# sentry/middleware/customer_domain.py -- __call__()
# DisallowedRedirect: Unsafe redirect exceeding 2048 characters

Root cause: The customer domain middleware redirects requests from org.sentry.io paths to canonical URLs. When the original URL has a very long path or query string, the redirect URL exceeds 2048 characters.

Actual fix: Resolved -- middleware now validates redirect URL length.

Example 3: MissingSchema on empty external URL (SENTRY-5E3V)

13,965 events | 95 users (resolved)

# sentry/net/http.py
# MissingSchema: Invalid URL '': No scheme supplied. Perhaps you meant https://?

Root cause: An integration or webhook configuration stores an empty string as the target URL. When the code attempts to make an HTTP request to this URL, the requests library raises MissingSchema.

Fix:

if not url or not url.startswith(("http://", "https://")):
    raise ValueError(f"Invalid URL: {url!r}")

Example 4: NoReverseMatch in URL construction (SENTRY-5G3B) -- unresolved

1,280 events | 120 users

In-app frames:

# django/urls/resolvers.py
# NoReverseMatch: Reverse for 'sentry-api-0-organization-group-group-events' with
# keyword arguments {'organization_id_or_slug': '...', ...} not found

Root cause: URL construction uses reverse() with keyword arguments that do not match the URL pattern. This can happen when URL patterns are changed but callers are not updated, or when the URL parameter values contain characters that don't match the pattern regex.

Fix:

try:
    url = reverse("sentry-api-0-organization-group-group-events", kwargs=kwargs)
except NoReverseMatch:
    logger.warning("url.reverse_failed", extra={"url_name": url_name, "kwargs": kwargs})
    url = None  # Or construct manually

Root Cause Analysis

Fix Patterns

Pattern A: Validate redirect URL length

def safe_redirect(request, url, max_length=2048):
    if len(url) > max_length:
        # Strip query string first
        parsed = urlparse(url)
        url = urlunparse(parsed._replace(query="", fragment=""))
        if len(url) > max_length:
            return HttpResponseBadRequest("URL too long")
    return redirect(url)

Pattern B: Validate URLs before HTTP requests

def validate_url(url):
    if not url:
        raise ValueError("Empty URL")
    if not url.startswith(("http://", "https://")):
        raise ValueError(f"Missing URL scheme: {url!r}")
    return url

Pattern C: Safe URL reversal

try:
    url = reverse(url_name, kwargs=kwargs)
except NoReverseMatch:
    logger.warning("url.no_reverse_match", extra={"name": url_name})
    url = fallback_url

Pattern D: Truncate query strings in redirects

def redirect_with_length_check(url):
    if len(url) <= 2048:
        return redirect(url)
    # Try without query string
    parsed = urlparse(url)
    base = f"{parsed.scheme}://{parsed.netloc}{parsed.path}"
    if len(base) <= 2048:
        return redirect(base)
    return HttpResponseBadRequest()

Detection Checklist

Scan the code for these patterns:

• [ ] Any redirect() or HttpResponseRedirect() -- is the URL length validated?
• [ ] Any middleware that constructs redirect URLs from the request path -- can the path be excessively long?
• [ ] Any HTTP request to a URL stored in the database -- is the URL validated for non-empty and proper scheme?
• [ ] Any reverse() call -- is NoReverseMatch handled?
• [ ] Any URL construction from user input (org slugs, project slugs) -- are invalid characters handled?
• [ ] Any redirect chain (middleware -> middleware) -- does each step validate the URL?