Exchange Session Detector

Exchange Session Detector

Production-grade pattern for detecting exchange trading sessions with full DST, holiday, and lunch break support. Validated in exness-data-preprocess across 10 global exchanges.

Self-Evolving Skill: This skill improves through use. If instructions are wrong, parameters drifted, or a workaround was needed — fix this file immediately, don't defer. Only update for real, reproducible issues.

When to Use

• Adding session flags (is_nyse_session, is_lse_session, etc.) to time-series DataFrames
• Detecting whether a timestamp falls within trading hours for any major exchange
• Checking for holidays (NYSE, LSE, or "major" when both are closed)
• Handling lunch breaks for Asian exchanges (Tokyo, Hong Kong, Singapore)
• Upgrading from simplified hour-range checks to production accuracy
• Building ClickHouse materialized columns for session classification

Architecture Overview

ExchangeConfig registry (exchanges.py)     SessionDetector (session_detector.py)
┌──────────────────────────────────┐      ┌──────────────────────────────────────┐
│ 10 frozen dataclasses            │      │ Wraps exchange_calendars library     │
│ ISO 10383 MIC codes              │─────▶│ Pre-computes trading minutes (sets)  │
│ IANA timezones for DST           │      │ Vectorized .isin() lookup (2.2x)    │
│ Local open/close hours           │      │ Holiday detection (NYSE + LSE)       │
└──────────────────────────────────┘      └──────────────────────────────────────┘

Quick Start

import exchange_calendars as xcals
import pandas as pd

# Single-exchange check
cal = xcals.get_calendar("XNYS")  # NYSE via ISO 10383 MIC
cal.is_open_on_minute(pd.Timestamp("2024-07-04 14:30", tz="UTC"))  # False (July 4th)
cal.is_open_on_minute(pd.Timestamp("2024-07-05 14:30", tz="UTC"))  # True

# Full session detection across 10 exchanges
from session_detector import SessionDetector
detector = SessionDetector()
df = detector.detect_sessions_and_holidays(dates_df)
# Adds: is_us_holiday, is_uk_holiday, is_major_holiday, is_{exchange}_session

The Two Tiers of Session Detection

Tier 1: Simple Hour-Range (What Most Projects Start With)

# Pattern from opendeviationbar-py/ouroboros.py
EXCHANGE_SESSION_HOURS = {
    "sydney":  {"tz": "Australia/Sydney",   "start": 10, "end": 16},
    "tokyo":   {"tz": "Asia/Tokyo",         "start":  9, "end": 15},
    "london":  {"tz": "Europe/London",      "start":  8, "end": 17},
    "newyork": {"tz": "America/New_York",   "start": 10, "end": 16},
}

def is_in_session(session_name, timestamp_utc):
    info = EXCHANGE_SESSION_HOURS[session_name]
    tz = zoneinfo.ZoneInfo(info["tz"])
    local_time = timestamp_utc.astimezone(tz)
    if local_time.weekday() >= 5:
        return False
    return info["start"] <= local_time.hour < info["end"]

What this gets right: DST conversion via zoneinfo, weekend exclusion.

What this misses:

• Holidays (Christmas, Thanksgiving, bank holidays)
• Lunch breaks (Tokyo 11:30-12:30, HK 12:00-13:00, SGX 12:00-13:00)
• Half-day / early close sessions
• Sub-hour precision (NYSE opens 9:30, not 10:00; LSE closes 16:30, not 17:00)
• Exchange schedule changes (Tokyo extended to 15:30 on Nov 5, 2024)

Tier 2: exchange_calendars (Production-Grade)

The exchange_calendars library (maintained, pip-installable, 50+ exchanges) handles all of the above automatically via is_open_on_minute(). The library uses IANA timezone data internally, so DST transitions are handled correctly without any manual logic.

Read references/exchange-registry.md for the full 10-exchange registry with MIC codes, timezones, and open/close hours.

Read references/session-detector-pattern.md for the complete SessionDetector implementation pattern with pre-computed trading minutes and vectorized lookup.

Exchange Registry

10 exchanges are supported via ISO 10383 MIC codes:

Adding a new exchange requires only one change: add an ExchangeConfig entry to the registry dict. The SessionDetector, schema generation, and column naming all propagate automatically.

Performance: Pre-Computed Trading Minutes

The naive approach calls calendar.is_open_on_minute() per timestamp per exchange — O(N \* E) with high constant factor. The validated pattern pre-computes all trading minutes into sets for O(1) lookup:

# Pre-compute once (startup cost, amortized over millions of lookups)
trading_minutes = detector._precompute_trading_minutes(start_date, end_date)
# Returns: {"nyse": {ts1, ts2, ...}, "lse": {ts1, ts2, ...}, ...}

# Vectorized lookup via pandas .isin() — 2.2x faster than per-row .apply()
df["is_nyse_session"] = df["ts"].isin(trading_minutes["nyse"]).astype(int)

The pre-computation itself uses is_open_on_minute() internally, so lunch breaks, holidays, and schedule changes are all respected.

Holiday Detection

# NYSE holidays (excludes weekends — only official closures)
nyse_holidays = {
    pd.to_datetime(h).date()
    for h in calendar.regular_holidays.holidays(start=start, end=end, return_name=False)
}

# Major holiday = both NYSE AND LSE closed
df["is_major_holiday"] = ((df["is_us_holiday"] == 1) & (df["is_uk_holiday"] == 1)).astype(int)

ClickHouse Integration

For server-side session detection (e.g., materialized columns), ClickHouse's toTimezone() handles DST automatically when given IANA timezone names:

-- DST-aware hour extraction (matches Python zoneinfo behavior)
ALTER TABLE my_table
UPDATE is_nyse_session = if(
    toHour(toTimezone(toDateTime(intDiv(close_time_ms, 1000)), 'America/New_York')) >= 9
    AND toHour(toTimezone(toDateTime(intDiv(close_time_ms, 1000)), 'America/New_York')) < 16
    AND toDayOfWeek(toTimezone(toDateTime(intDiv(close_time_ms, 1000)), 'America/New_York')) <= 5,
    1, 0
) WHERE 1 = 1

Limitation: ClickHouse toTimezone() handles DST but not holidays or lunch breaks. For those, compute in Python and write the flags back, or maintain a holiday calendar table in ClickHouse.

Upgrade Path: Hour-Range to exchange_calendars

1. pip install exchange_calendars (or add to pyproject.toml) 2. Replace fixed-hour dicts with ExchangeConfig registry (see references/exchange-registry.md) 3. Replace zoneinfo hour checks with SessionDetector.detect_sessions_and_holidays() 4. Update tests to cover: holidays, lunch breaks, DST transitions, early closes

The exchange_calendars library is ~10MB installed and has no heavy dependencies beyond pandas and numpy. Calendar data is bundled (no network calls at runtime).

References

Source

Validated implementation: ~/eon/exness-data-preprocess/src/exness_data_preprocess/session_detector.py + exchanges.py

Simplified predecessor: ~/eon/opendeviationbar-py/python/opendeviationbar/ouroboros.py (Tier 1 only)

Post-Execution Reflection

After this skill completes, check before closing:

1. Did the command succeed? — If not, fix the instruction or error table that caused the failure. 2. Did parameters or output change? — If the underlying tool's interface drifted, update Usage examples and Parameters table to match. 3. Was a workaround needed? — If you had to improvise (different flags, extra steps), update this SKILL.md so the next invocation doesn't need the same workaround.

Only update if the issue is real and reproducible — not speculative.

ClickHouse Session Detection SQL

Server-side session detection using ClickHouse's toTimezone() for DST-aware hour extraction.

How It Works

ClickHouse's toTimezone() accepts IANA timezone names and handles DST transitions automatically, matching Python's zoneinfo behavior. This means the same session logic can run server-side without round-tripping data to Python.

SQL Pattern: ALTER TABLE UPDATE

Used by opendeviationbar-py to backfill session columns on existing data:

_SESSION_UPDATES = [
    {"column": "exchange_session_sydney",  "tz": "Australia/Sydney",  "start": "10", "end": "16"},
    {"column": "exchange_session_tokyo",   "tz": "Asia/Tokyo",        "start": "9",  "end": "15"},
    {"column": "exchange_session_london",  "tz": "Europe/London",     "start": "8",  "end": "17"},
    {"column": "exchange_session_newyork", "tz": "America/New_York",  "start": "10", "end": "16"},
]

def _build_session_update_sql(session, *, symbol=None):
    col, tz = session["column"], session["tz"]
    start, end = session["start"], session["end"]

    ts_local = f"toTimezone(toDateTime(intDiv(close_time_ms, 1000)), '{tz}')"
    condition = (
        f"toHour({ts_local}) >= {start} "
        f"AND toHour({ts_local}) < {end} "
        f"AND toDayOfWeek({ts_local}) <= 5"
    )

    where = f"symbol = '{symbol}'" if symbol else "1 = 1"

    return (
        f"ALTER TABLE opendeviationbar_cache.open_deviation_bars "
        f"UPDATE {col} = if({condition}, 1, 0) "
        f"WHERE {where}"
    )

Generated SQL Example

ALTER TABLE opendeviationbar_cache.open_deviation_bars
UPDATE exchange_session_newyork = if(
    toHour(toTimezone(toDateTime(intDiv(close_time_ms, 1000)), 'America/New_York')) >= 10
    AND toHour(toTimezone(toDateTime(intDiv(close_time_ms, 1000)), 'America/New_York')) < 16
    AND toDayOfWeek(toTimezone(toDateTime(intDiv(close_time_ms, 1000)), 'America/New_York')) <= 5,
    1, 0
) WHERE 1 = 1

Limitations vs Python exchange_calendars

For production accuracy with holidays and lunch breaks, the recommended approach is:

1. Compute session flags in Python using exchange_calendars 2. Write flags back to ClickHouse as columns 3. Use ClickHouse toTimezone() only for simple hour-range checks where holidays don't matter

Holiday Table Pattern (Advanced)

If you need server-side holiday detection without Python:

-- Create holiday calendar table
CREATE TABLE exchange_holidays (
    exchange String,
    holiday_date Date,
    holiday_name String
) ENGINE = MergeTree()
ORDER BY (exchange, holiday_date);

-- Insert holidays from exchange_calendars (one-time Python script)
-- Then join in queries:
SELECT *
FROM my_table t
LEFT JOIN exchange_holidays h
    ON h.exchange = 'NYSE'
    AND h.holiday_date = toDate(toTimezone(toDateTime(intDiv(t.close_time_ms, 1000)), 'America/New_York'))
WHERE h.holiday_date IS NULL  -- Not a holiday

Source

Canonical: ~/eon/opendeviationbar-py/python/opendeviationbar/clickhouse/migrations.py

Exchange Registry Pattern

Frozen dataclass registry for exchange configuration. Single source of truth — adding a new exchange requires only one dict entry.

ExchangeConfig Dataclass

from dataclasses import dataclass
from typing import Dict


@dataclass(frozen=True)
class ExchangeConfig:
    """
    Immutable configuration for a single exchange.

    Attributes:
        code: ISO 10383 MIC code (e.g., "XNYS" for NYSE)
        name: Full exchange name
        currency: Primary currency
        timezone: IANA timezone (DST handled by exchange_calendars)
        country: Country name
        open_hour: Trading start hour in local time (24h)
        open_minute: Trading start minute
        close_hour: Trading close hour in local time (24h)
        close_minute: Trading close minute
    """
    code: str
    name: str
    currency: str
    timezone: str
    country: str
    open_hour: int
    open_minute: int
    close_hour: int
    close_minute: int

Registry (10 Exchanges)

EXCHANGES: Dict[str, ExchangeConfig] = {
    "nyse": ExchangeConfig(
        code="XNYS", name="New York Stock Exchange",
        currency="USD", timezone="America/New_York", country="United States",
        open_hour=9, open_minute=30, close_hour=16, close_minute=0,
    ),
    "lse": ExchangeConfig(
        code="XLON", name="London Stock Exchange",
        currency="GBP", timezone="Europe/London", country="United Kingdom",
        open_hour=8, open_minute=0, close_hour=16, close_minute=30,
    ),
    "xswx": ExchangeConfig(
        code="XSWX", name="SIX Swiss Exchange",
        currency="CHF", timezone="Europe/Zurich", country="Switzerland",
        open_hour=9, open_minute=0, close_hour=17, close_minute=30,
    ),
    "xfra": ExchangeConfig(
        code="XFRA", name="Frankfurt Stock Exchange",
        currency="EUR", timezone="Europe/Berlin", country="Germany",
        open_hour=9, open_minute=0, close_hour=17, close_minute=30,
    ),
    "xtse": ExchangeConfig(
        code="XTSE", name="Toronto Stock Exchange",
        currency="CAD", timezone="America/Toronto", country="Canada",
        open_hour=9, open_minute=30, close_hour=16, close_minute=0,
    ),
    "xnze": ExchangeConfig(
        code="XNZE", name="New Zealand Exchange",
        currency="NZD", timezone="Pacific/Auckland", country="New Zealand",
        open_hour=10, open_minute=0, close_hour=16, close_minute=45,
    ),
    "xtks": ExchangeConfig(
        code="XTKS", name="Tokyo Stock Exchange",
        currency="JPY", timezone="Asia/Tokyo", country="Japan",
        open_hour=9, open_minute=0, close_hour=15, close_minute=0,
        # Lunch break: 11:30-12:30 JST (handled by exchange_calendars)
    ),
    "xasx": ExchangeConfig(
        code="XASX", name="Australian Securities Exchange",
        currency="AUD", timezone="Australia/Sydney", country="Australia",
        open_hour=10, open_minute=0, close_hour=16, close_minute=0,
    ),
    "xhkg": ExchangeConfig(
        code="XHKG", name="Hong Kong Stock Exchange",
        currency="HKD", timezone="Asia/Hong_Kong", country="Hong Kong",
        open_hour=9, open_minute=30, close_hour=16, close_minute=0,
        # Lunch break: 12:00-13:00 HKT (handled by exchange_calendars)
    ),
    "xses": ExchangeConfig(
        code="XSES", name="Singapore Exchange",
        currency="SGD", timezone="Asia/Singapore", country="Singapore",
        open_hour=9, open_minute=0, close_hour=17, close_minute=0,
        # Lunch break: 12:00-13:00 SGT (handled by exchange_calendars)
    ),
}

Helper Functions

def get_exchange_names() -> list[str]:
    """Get all registry keys: ["nyse", "lse", "xswx", ...]"""
    return list(EXCHANGES.keys())

def get_exchange_config(name: str) -> ExchangeConfig:
    """Lookup by name. Raises ValueError with available list on miss."""
    if name not in EXCHANGES:
        available = ", ".join(EXCHANGES.keys())
        raise ValueError(f"Unknown exchange: {name}. Available: {available}")
    return EXCHANGES[name]

Adding a New Exchange

1. Find the ISO 10383 MIC code (e.g., XBOM for BSE India) 2. Verify exchange_calendars supports it: xcals.get_calendar("XBOM") 3. Add one entry to EXCHANGES dict 4. Everything else propagates: SessionDetector picks it up, columns are named is_xbom_session

Source

Canonical: ~/eon/exness-data-preprocess/src/exness_data_preprocess/exchanges.py

SessionDetector Pattern

Complete implementation pattern for DST-aware, holiday-aware, lunch-break-aware session detection.

Class Structure

"""
Exchange calendar operations and session/holiday detection.

Uses exchange_calendars library to determine trading hours, holidays, and
lunch breaks for 10 global exchanges.

Handles:
- Exchange calendar initialization from EXCHANGES registry
- Holiday detection for NYSE and LSE (official closures only)
- Major holiday detection (both NYSE and LSE closed)
- Trading session detection with lunch break support
  (Tokyo 11:30-12:30 JST, Hong Kong 12:00-13:00 HKT, Singapore 12:00-13:00 SGT)

Performance:
- Pre-computes trading minutes for vectorized lookup (2.2x speedup)
- Preserves accuracy via exchange_calendars.is_open_on_minute()
"""

from datetime import date
from typing import Any, Dict, Set

import exchange_calendars as xcals
import pandas as pd

from exchanges import EXCHANGES


class SessionDetector:
    """
    Detect trading sessions and holidays for global exchanges.

    Lunch Breaks (automatically handled by exchange_calendars):
    - Tokyo (XTKS): 11:30-12:30 JST
    - Hong Kong (XHKG): 12:00-13:00 HKT
    - Singapore (XSES): 12:00-13:00 SGT
    """

    def __init__(self):
        self.calendars: Dict[str, Any] = {}
        for exchange_name, exchange_config in EXCHANGES.items():
            self.calendars[exchange_name] = xcals.get_calendar(exchange_config.code)

    def _precompute_trading_minutes(
        self, start_date: date, end_date: date
    ) -> Dict[str, Set[pd.Timestamp]]:
        """
        Pre-compute trading minutes for all exchanges in date range.

        Returns dict mapping exchange_name to set of trading minutes
        (timezone-aware UTC timestamps). Enables vectorized .isin() lookup.

        Uses calendar.is_open_on_minute() during pre-computation to respect:
        - Lunch breaks (Tokyo, Hong Kong, Singapore)
        - Trading hour changes (e.g., Tokyo extended to 15:30 on Nov 5, 2024)
        - Holidays and weekends (automatically excluded)
        """
        trading_minutes: Dict[str, Set[pd.Timestamp]] = {}

        for exchange_name, calendar in self.calendars.items():
            minutes_set: Set[pd.Timestamp] = set()

            sessions = calendar.sessions_in_range(start_date, end_date)

            for session_date in sessions:
                market_open = calendar.session_open(session_date)
                market_close = calendar.session_close(session_date)

                current_minute = market_open
                while current_minute <= market_close:
                    if calendar.is_open_on_minute(current_minute):
                        minutes_set.add(current_minute)
                    current_minute += pd.Timedelta(minutes=1)

            trading_minutes[exchange_name] = minutes_set

        return trading_minutes

    def detect_sessions_and_holidays(self, dates_df: pd.DataFrame) -> pd.DataFrame:
        """
        Add holiday and session columns to dates DataFrame.

        Args:
            dates_df: DataFrame with 'ts' column (timezone-aware UTC) and 'date' column

        Returns:
            DataFrame with added columns:
                - is_us_holiday: 1 if NYSE closed (excludes weekends)
                - is_uk_holiday: 1 if LSE closed (excludes weekends)
                - is_major_holiday: 1 if both NYSE and LSE closed
                - is_{exchange}_session: 1 if during trading hours (excludes lunch)
        """
        start_date = dates_df["ts"].min().date()
        end_date = dates_df["ts"].max().date()

        # Holiday detection — NYSE and LSE only
        nyse_holidays = {
            pd.to_datetime(h).date()
            for h in self.calendars["nyse"].regular_holidays.holidays(
                start=start_date, end=end_date, return_name=False
            )
        }
        lse_holidays = {
            pd.to_datetime(h).date()
            for h in self.calendars["lse"].regular_holidays.holidays(
                start=start_date, end=end_date, return_name=False
            )
        }

        dates_df["is_us_holiday"] = dates_df["ts"].dt.date.apply(
            lambda d: int(d in nyse_holidays)
        )
        dates_df["is_uk_holiday"] = dates_df["ts"].dt.date.apply(
            lambda d: int(d in lse_holidays)
        )
        dates_df["is_major_holiday"] = (
            (dates_df["is_us_holiday"] == 1) & (dates_df["is_uk_holiday"] == 1)
        ).astype(int)

        # Session detection — pre-compute then vectorize
        trading_minutes = self._precompute_trading_minutes(start_date, end_date)

        for exchange_name in self.calendars.keys():
            col_name = f"is_{exchange_name}_session"
            dates_df[col_name] = (
                dates_df["ts"].isin(trading_minutes[exchange_name]).astype(int)
            )

        return dates_df

Usage Example

import pandas as pd

detector = SessionDetector()

# Build a DataFrame with minute-level timestamps
dates_df = pd.DataFrame({
    "ts": pd.date_range("2024-01-01", "2024-12-31", freq="1min", tz="UTC"),
})
dates_df["date"] = dates_df["ts"].dt.date

result = detector.detect_sessions_and_holidays(dates_df)

# Result columns:
# is_us_holiday, is_uk_holiday, is_major_holiday,
# is_nyse_session, is_lse_session, is_xswx_session, is_xfra_session,
# is_xtse_session, is_xnze_session, is_xtks_session, is_xasx_session,
# is_xhkg_session, is_xses_session

Performance Notes

• Pre-computation cost: ~2-5 seconds for 1 year of data across 10 exchanges
• Lookup cost: O(1) per timestamp via set .isin()
• Speedup: 2.2x vs per-timestamp .apply(calendar.is_open_on_minute)
• Memory: ~500K timestamps per exchange per year (trading minutes only)

Key API from exchange_calendars

cal = xcals.get_calendar("XNYS")

# Session queries
cal.sessions_in_range(start_date, end_date)  # Trading days (DatetimeIndex)
cal.session_open(session_date)               # Market open (Timestamp, UTC)
cal.session_close(session_date)              # Market close (Timestamp, UTC)
cal.is_open_on_minute(timestamp)             # Respects lunch breaks + holidays

# Holiday queries
cal.regular_holidays.holidays(start, end)    # Official closure dates
cal.adhoc_holidays                           # One-off closures

Source

Canonical: ~/eon/exness-data-preprocess/src/exness_data_preprocess/session_detector.py