Pikud Haoref Alerts

# Alternative Data Sources

## Tzofar (tzevaadom.co.il)

Tzofar is a popular third-party alert relay service. Unlike the official oref.org.il API, Tzofar endpoints are **not geo-blocked** — they work from any IP worldwide.

### Endpoints

**Required headers:** Tzofar blocks Python's default `User-Agent: Python-urllib/X.Y` with HTTP 403. Set a browser-like User-Agent:

```python
# Works
requests.get(url)  # requests uses its own UA, not blocked
urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})

# Fails with 403
urllib.request.urlopen(url)  # sends Python-urllib/3.x
```

**Recent alert groups (last 50):**
```
GET https://api.tzevaadom.co.il/alerts-history
```

Returns the last 50 alert groups. No pagination support found.

**Single alert group by ID:**
```
GET https://api.tzevaadom.co.il/alerts-history/id/{id}
```

**Historical data via ID iteration:**

There is no bulk download endpoint. To build historical data, iterate alert group IDs backwards from the latest known ID:

```
GET https://api.tzevaadom.co.il/alerts-history/id/5913
GET https://api.tzevaadom.co.il/alerts-history/id/5912
GET https://api.tzevaadom.co.il/alerts-history/id/5911
...
```

**Rate limiting:** Tzofar rate-limits burst traffic — rapid-fire requests with no delay trigger HTTP 429 after ~13 requests. Adding short delays (0.3–0.5 seconds between requests) avoids 429s entirely and allows fetching hundreds of groups in a few minutes.

The `/alerts-history` endpoint returns the most recent ~50 groups — use the lowest `id` from that response as your starting point for backward iteration. **IDs are mostly sequential but gaps exist** — e.g., IDs 5599–5663 all return 404 while surrounding IDs work. When iterating, skip 404s and stop after a tolerance limit (e.g., 100 consecutive 404s). **Optimization for large gaps:** if you hit 10+ consecutive 404s, try jumping back by 50 IDs to skip past the gap faster, then backfill if needed.

**Ready-to-use iteration function (stdlib only):**

```python
import urllib.request, json, time

def fetch_tzofar_history(start_id, min_date_unix=0, max_gap=100, delay=0.35):
    """Iterate Tzofar alert group IDs backwards.

    Args:
        start_id: Highest ID to start from (get from /alerts-history).
        min_date_unix: Stop when alerts are older than this Unix timestamp.
        max_gap: Stop after this many consecutive 404s.
        delay: Seconds between requests (0.3-0.5 avoids 429s).

    Returns: List of alert group dicts, newest first.
    """
    results = []
    consecutive_404s = 0
    current_id = start_id

    while consecutive_404s < max_gap and current_id > 0:
        req = urllib.request.Request(
            f"https://api.tzevaadom.co.il/alerts-history/id/{current_id}",
            headers={"User-Agent": "Mozilla/5.0"}
        )
        try:
            with urllib.request.urlopen(req, timeout=10) as resp:
                group = json.loads(resp.read())
                consecutive_404s = 0
                # Check if oldest alert in group is before our cutoff
                times = [a["time"] for a in group.get("alerts", [])]
                if times and min(times) < min_date_unix:
                    results.append(group)
                    break
                results.append(group)
        except urllib.error.HTTPError as e:
            if e.code == 404:
                consecutive_404s += 1
            elif e.code == 429:
                time.sleep(2)  # back off on rate limit
                continue  # retry same ID
            else:
                raise
        current_id -= 1
        time.sleep(delay)

    return results
```

**Expected throughput:** ~200 groups in ~70 seconds at 0.35s pacing. Plan for 2+ minutes when fetching 300+ groups (e.g., a full week of conflict). If running inside an LLM agent with bash timeouts, consider splitting into batches with intermediate saves.

### Data model

Tzofar groups alerts into "alert groups." Each group represents an **incident** (typically a single