Django Celery Expert

# Configuration Guide

## Django-Celery Integration Setup

### Project Structure

```
myproject/
├── myproject/
│   ├── __init__.py
│   ├── celery.py      # Celery app configuration
│   ├── settings.py
│   └── urls.py
├── myapp/
│   ├── __init__.py
│   ├── models.py
│   └── tasks.py       # App-specific tasks
└── manage.py
```

### Celery Application Setup

```python
# myproject/celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')

app = Celery('myproject')

# Load config from Django settings with CELERY_ prefix
app.config_from_object('django.conf:settings', namespace='CELERY')

# Auto-discover tasks in all installed apps
app.autodiscover_tasks()

@app.task(bind=True, ignore_result=True)
def debug_task(self):
    print(f'Request: {self.request!r}')
```

```python
# myproject/__init__.py
from .celery import app as celery_app

__all__ = ('celery_app',)
```

### Django Settings

```python
# settings.py

# Broker Configuration (choose one)
# Redis
CELERY_BROKER_URL = 'redis://localhost:6379/0'

# RabbitMQ
CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672//'

# Result Backend (optional, needed if you track results)
CELERY_RESULT_BACKEND = 'redis://localhost:6379/1'
# Or use Django database
CELERY_RESULT_BACKEND = 'django-db'

# Serialization
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_ACCEPT_CONTENT = ['json']

# Timezone
CELERY_TIMEZONE = 'UTC'
CELERY_ENABLE_UTC = True

# Task settings
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 30 * 60  # 30 minutes hard limit
CELERY_TASK_SOFT_TIME_LIMIT = 25 * 60  # 25 minutes soft limit
```

## Broker Configuration

### Redis Broker

```python
# Basic Redis
CELERY_BROKER_URL = 'redis://localhost:6379/0'

# Redis with password
CELERY_BROKER_URL = 'redis://:password@localhost:6379/0'

# Redis Sentinel for HA
CELERY_BROKER_URL = 'sentinel://localhost:26379'
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'master_name': 'mymaster',
    'sentinel_kwargs': {'password': 'sentinel_password'},
}

# Redis Cluster
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'cluster': True,
}

# Connection pool settings
CELERY_BROKER_POOL_LIMIT = 10
CELERY_BROKER_CONNECTION_TIMEOUT = 10
CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True
```

### RabbitMQ Broker

```python
# Basic RabbitMQ
CELERY_BROKER_URL = 'amqp://guest:guest@localhost:5672//'

# With virtual host
CELERY_BROKER_URL = 'amqp://user:password@localhost:5672/myvhost'

# RabbitMQ with SSL
CELERY_BROKER_URL = 'amqp://user:password@localhost:5671//'
CELERY_BROKER_USE_SSL = {
    'keyfile': '/path/to/key.pem',
    'certfile': '/path/to/cert.pem',
    'ca_certs': '/path/to/ca.pem',
    'cert_reqs': ssl.CERT_REQUIRED,
}

# Connection pool
CELERY_BROKER_POOL_LIMIT = 10
CELERY_BROKER_HEARTBEAT = 10

# Confirm message delivery (recommended for reliability)
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'confirm_publish': True,
}
```

## Broker Transport Options

Critical settings for message delivery reliability.

### Redis Visibility Timeout

When using Redis, set `visibility_timeout` to control how long a task remains invisible after being picked up. If a worker crashes before acknowledging, the task becomes visible again after this timeout.

```python
# Redis visibility timeout (default: 1 hour)
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'visibility_timeout': 3600,  # 1 hour in seconds
}
```

**Important:** Set visibility timeout longer than your longest task. If a task takes longer than the timeout, it will be redelivered to another worker while still running.

### RabbitMQ Publisher Confirms

Enable publisher confirms to ensure messages are actually delivered to the broker.

```python
# RabbitMQ: confirm message delivery
CELERY_BROKER_TRANSPORT_OPTIONS = {
    'confirm_publish': True,
}
```

### SQS Configuration

```python
# AWS SQS
CELERY_BROKER_URL = 'sqs://aws_access_key:aws_secret_key@'
CELERY_BROKER_TRANSPORT_OPTIONS = {