Data Specification¶

Data Sources¶

Data Fields¶

OHLCV Standard Schema¶

Extended Fields (When Available)¶

FX Sessions¶

Session Definitions (UTC)¶

Kill Zones (High-Probability Windows)¶

Session Overlap Handling¶

def get_active_sessions(timestamp_utc):
    """Return list of active sessions for given UTC timestamp."""
    hour = timestamp_utc.hour
    sessions = []
    if 21 <= hour or hour < 6:
        sessions.append('sydney')
    if 0 <= hour < 9:
        sessions.append('tokyo')
    if 7 <= hour < 16:
        sessions.append('london')
    if 12 <= hour < 21:
        sessions.append('new_york')
    return sessions

Crypto: 24/7 Handling¶

Funding Rate Schedule¶

Funding Rate Integration¶

# data/catalog.yaml entry
btc_perpetual:
  symbol: BTCUSDT
  type: perpetual
  funding_rate_field: funding_rate
  funding_times_utc: ["00:00", "08:00", "16:00"]
  include_funding_in_backtest: true

Weekend/Holiday Handling¶

• Crypto trades 24/7/365 - no gaps expected
• Reduced liquidity: Saturday 00:00 - Sunday 12:00 UTC
• Flag low-liquidity periods in data catalog
• Widen slippage assumptions during low-liquidity

Futures Rollovers¶

Contract Specifications¶

Month Codes¶

Rollover Protocol¶

def build_continuous_contract(symbol_root, roll_days_before=8):
    """
    Build continuous contract series using back-adjustment.

    Method: Back-adjust prior prices by roll gap
    - Preserves percentage returns
    - No lookahead (adjustment applied to past, not future)
    """
    pass  # Implementation in src/data/rollovers.py

Rollover Data Schema¶

Corporate Actions (Indices)¶

Handling Policy¶

Note¶

For index futures (NQ, ES), corporate actions are absorbed into the index calculation. No manual adjustment required. Data from TradingView is pre-adjusted.

Data Quality Rules¶

Validation Checks¶

Gap Handling¶

def handle_gaps(df, max_gap_bars=5):
    """
    Handle missing bars in OHLCV data.

    Rules:
    - Gap <= max_gap_bars: Forward-fill close, zero volume
    - Gap > max_gap_bars: Mark as session break, don't fill
    - Weekend gaps: Expected, mark as session_break=True
    """
    pass  # Implementation in src/data/validators.py

Data Versioning¶

Version Control Strategy¶

data/
├── raw/
│   └── {symbol}/
│       └── {timeframe}/
│           └── {YYYY}/{MM}/
│               └── {symbol}_{timeframe}_{YYYYMMDD}.parquet
│
├── processed/
│   └── v{VERSION}/
│       └── {symbol}_{timeframe}.parquet
│
└── catalog.yaml  # Tracks versions and lineage

Catalog Schema¶

# data/catalog.yaml
version: "2024.01.15"

datasets:
  eurusd_1h:
    symbol: EURUSD
    timeframe: 1H
    source: tradingview
    start_date: "2020-01-01"
    end_date: "2024-01-15"
    row_count: 35040
    checksum: sha256:abc123...
    processed_version: v3
    processing_date: "2024-01-15"
    quality_report:
      missing_bars: 0.2%
      flagged_bars: 12

  btcusdt_15m:
    symbol: BTCUSDT
    timeframe: 15m
    source: binance
    start_date: "2021-01-01"
    end_date: "2024-01-15"
    row_count: 105120
    checksum: sha256:def456...
    includes_funding: true

Immutability Rules¶

1. Raw data is append-only (never modified)
2. Processed data is versioned (new version on reprocessing)
3. Catalog tracks all versions and lineage
4. Checksums verify data integrity
5. Git LFS for large files (> 50MB)

Data Refresh Schedule¶

Fetch Commands¶

# Fetch recent data
python -m src.data.fetch --symbol EURUSD --timeframe 1H --days 30

# Validate existing data
python -m src.data.validate --dataset eurusd_1h

# Build continuous futures
python -m src.data.rollovers --symbol NQ --method back_adjusted

# Update catalog checksums
python -m src.data.catalog --update-checksums