# CLAUDE.md - Project Memory

**opendeviationbar-py**: Rust workspace with Python bindings via PyO3/maturin. Publishes to PyPI (`opendeviationbar`) and crates.io (`opendeviationbar-core`, `opendeviationbar-providers`, `opendeviationbar-streaming`).

> **Naming**: This project was renamed from `rangebar-py` → `opendeviationbar-py` in March 2026. The algorithm is NOT a classic Nicolellis range bar (which uses dynamic `High - Low` range). It uses **open-anchored deviation thresholds**: breach when `price >= open * (1 + threshold)` or `price <= open * (1 - threshold)`. This is a genuinely novel bar type — an open-anchored, bidirectional, percentage-threshold bar. The name "Open Deviation Bar" precisely describes the mechanism. The legacy `rangebar` PyPI package and `rangebar-core`/`rangebar-providers`/`rangebar-streaming` crates.io packages redirect to the new names via shim releases.

---

## Quick Reference

| Task                         | Entry Point                                                                                             | Details                                               |
| ---------------------------- | ------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
| Generate open deviation bars | `get_open_deviation_bars()`                                                                             | [Python API](#python-api)                             |
| Real-time streaming          | `run_sidecar()` / `scripts/streaming_sidecar.py`                                                        | [ADR](/docs/adr/2026-01-31-realtime-streaming-api.md) |
| Understand architecture      | [docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md)                                                           | 9-crate workspace                                     |
| Work with Rust crates        | [crates/CLAUDE.md](/crates/CLAUDE.md)                                                                   | Core algorithm, microstructure, inter-bar features    |
| Work with Python layer       | [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md)                                 | API, caching, validation, symbol registry             |
| ClickHouse cache ops         | [python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md)           | Schema, population, remote setup, dedup hardening     |
| Plugin system                | [python/opendeviationbar/plugins/CLAUDE.md](/python/opendeviationbar/plugins/CLAUDE.md)                 | FeatureProvider protocol, entry-point discovery       |
| Operations & scripts         | [scripts/CLAUDE.md](/scripts/CLAUDE.md)                                                                 | Pueue, cache population, per-year parallelism         |
| Release workflow             | [docs/development/RELEASE.md](/docs/development/RELEASE.md)                                             | Zig cross-compile, mise tasks                         |
| Deploy to bigblack           | `mise run deploy:bigblack`                                                                              | Git pull + PyPI install + verify                      |
| Performance monitoring       | [docs/development/PERFORMANCE.md](/docs/development/PERFORMANCE.md)                                     | Benchmarks, metrics                                   |
| Performance optimization     | [.claude/pgo-workflow.md](/.claude/pgo-workflow.md)                                                     | LTO, PGO, SIMD, GPU strategies (Issue #96)            |
| Streaming sidecar (P0)       | [docs/development/ISSUE-107-SIDECAR-RELIABILITY.md](/docs/development/ISSUE-107-SIDECAR-RELIABILITY.md) | Watchdog, error recovery, systemd auto-restart        |
| Project context              | [docs/CONTEXT.md](/docs/CONTEXT.md)                                                                     | Why this project exists                               |
| API reference                | [docs/api/INDEX.md](/docs/api/INDEX.md)                                                                 | Full Python API docs                                  |
| Research                     | [docs/research/INDEX.md](/docs/research/INDEX.md)                                                       | ML labeling, regime patterns, TDA                     |
| Oracle verification          | [docs/verification/](/docs/verification/)                                                               | Bit-exact cross-reference reports                     |
| Source validation            | [.claude/skills/source-validation/SKILL.md](/.claude/skills/source-validation/SKILL.md)                 | Vision→REST→WS junction proof, fromId pagination      |
| Deploy secrets (local.env)   | `deploy/opendeviationbar.local.env`                                                                     | Survives git reset, holds tokens                      |
| Heartbeat system             | `scripts/health_heartbeat.py`                                                                           | Ephemeral Telegram heartbeat via systemd              |
| Autonomous agent ops         | [.claude/skills/autonomous-agent-ops/SKILL.md](/.claude/skills/autonomous-agent-ops/SKILL.md)           | Multi-agent monitoring, pueue guardrails, watchdogs   |

---

## Critical Principles

### 1. Leverage Rust

**ALWAYS use local Rust crates for heavy lifting.** Python handles I/O only.

1. **Check Rust first**: Before writing Python code, check if `opendeviationbar-core` already provides the capability
2. **Stream to Rust**: The `OpenDeviationBarProcessor` maintains state between `process_trades()` calls
3. **Checkpoint API**: Use `create_checkpoint()` and `from_checkpoint()` for cross-session continuity
4. **No reinventing**: Don't reimplement open deviation bar logic in Python

```python
# CORRECT: Stream to Rust (maintains state automatically)
for chunk in data_stream:
    bars = processor.process_trades(chunk.to_dicts())

# WRONG: Buffer in Python (OOM risk)
all_data = []
for chunk in data_stream:
    all_data.extend(chunk)
```

### 2. Per-Year Parallelization (Default for Cache Population)

**Never queue a monolithic multi-year cache population job.** Ouroboros resets processor state at year boundaries, making each year an independent unit. Always split into per-year pueue jobs.

- A single DOGEUSDT@500 job: **22 days**. Per-year splits: **3-4 days**.
- Each year gets its own checkpoint file, ClickHouse dedup handles overlaps.
- **Full details**: [scripts/CLAUDE.md](/scripts/CLAUDE.md#per-year-parallelization-default-strategy)
- **Skills**: `Skill(devops-tools:pueue-job-orchestration)` | `Skill(devops-tools:distributed-job-safety)`

### 3. Symbol Registry Gate

**Every symbol must be registered before processing.** Unregistered symbols raise `SymbolNotRegisteredError`.

- **SSoT**: `symbols.toml` (repo root symlink → `python/opendeviationbar/data/symbols.toml`)
- **Adding new symbols**: Edit `symbols.toml`, run `maturin develop`, then process
- **Override** (dev only): `export OPENDEVIATIONBAR_SYMBOL_GATE=off`
- **Full details**: [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md#symbol-registry-issue-79)

---

## Python API

```python
from opendeviationbar import get_open_deviation_bars, get_n_open_deviation_bars, process_trades_polars

# Date-bounded (backtesting)
df = get_open_deviation_bars("BTCUSDT", "2024-01-01", "2024-06-30")

# Count-bounded (ML training)
df = get_n_open_deviation_bars("BTCUSDT", n_bars=10000)

# Polars users (2-3x faster)
import polars as pl
trades = pl.scan_parquet("trades.parquet")
df = process_trades_polars(trades, threshold_decimal_bps=250)

# With microstructure features
df = get_open_deviation_bars("BTCUSDT", "2024-01-01", "2024-06-30", include_microstructure=True)

# Long ranges (>30 days) — must populate cache first
from opendeviationbar import populate_cache_resumable
populate_cache_resumable("BTCUSDT", "2019-01-01", "2025-12-31")
df = get_open_deviation_bars("BTCUSDT", "2019-01-01", "2025-12-31")
```

| API                           | Use Case                   | Details                                                   |
| ----------------------------- | -------------------------- | --------------------------------------------------------- |
| `get_open_deviation_bars()`   | Date range, backtesting    | [docs/api/primary-api.md](/docs/api/primary-api.md)       |
| `get_n_open_deviation_bars()` | Exact N bars, ML           | [docs/api/primary-api.md](/docs/api/primary-api.md)       |
| `populate_cache_resumable()`  | Long ranges (>30 days)     | [docs/api/cache-api.md](/docs/api/cache-api.md)           |
| `process_trades_polars()`     | Polars DataFrames          | [docs/api/processing-api.md](/docs/api/processing-api.md) |
| `process_trades_chunked()`    | Large datasets >10M trades | [docs/api/processing-api.md](/docs/api/processing-api.md) |

---

## Architecture

```
opendeviationbar-py/
├── crates/                    9 Rust crates (3 on crates.io, 6 internal)
│   └── opendeviationbar-core/         Core algorithm, microstructure features
├── src/                       PyO3 bindings (Issue #94: domain modules)
│   ├── lib.rs                 Thin orchestrator (~200 lines)
│   ├── helpers.rs             Conversion functions
│   ├── core_bindings.rs       PyOpenDeviationBarProcessor, PyPositionVerification
│   ├── arrow_bindings.rs      Arrow export functions
│   ├── binance_bindings.rs    Binance data fetching
│   ├── exness_bindings.rs     Exness data fetching
│   ├── streaming_bindings.rs  LiveBarEngine, streaming classes
│   └── stream_manager_bindings.rs  StreamManager (Issue #161)
├── python/opendeviationbar/           Python API layer
│   ├── clickhouse/            ClickHouse cache (bigblack)
│   ├── sidecar.py             Streaming sidecar orchestrator (v12.20+)
│   ├── validation/            Microstructure validation
│   └── storage/               Tier 1 cache (Parquet)
├── scripts/                   Pueue jobs, validation, cache population
├── .cargo/config.toml         Cross-compile friendly rustflags
└── pyproject.toml             Maturin config
```

**Key files**: `src/lib.rs` (PyO3 module registration), `python/opendeviationbar/__init__.py` (public API), `crates/opendeviationbar-core/src/processor.rs` (core algorithm), `python/opendeviationbar/sidecar.py` (streaming sidecar)

**Full architecture**: [docs/ARCHITECTURE.md](/docs/ARCHITECTURE.md)

---

## Development Commands

```bash
# Setup (mise manages all tools including zig)
mise install

# Build & test
mise run build              # maturin develop
mise run test               # Rust tests (cargo nextest)
mise run test-py            # Python tests (pytest)

# Quality
mise run check-full         # fmt + lint + test + deny

# Release (see docs/development/RELEASE.md)
mise run release:full       # Full 4-phase workflow
mise run release:linux      # Zig cross-compile (~55 sec)
mise run release:pypi         # Upload to PyPI

# Benchmarks
mise run bench:run          # Full benchmarks
mise run bench:validate     # Verify 1M ticks < 100ms
```

---

## Build System

Zig cross-compilation from macOS (no remote SSH needed).

| Platform     | Strategy          | Time    | Command                        |
| ------------ | ----------------- | ------- | ------------------------------ |
| macOS ARM64  | Native maturin    | ~10 sec | `mise run release:macos-arm64` |
| Linux x86_64 | Zig cross-compile | ~55 sec | `mise run release:linux`       |

**Why rustls**: Eliminates OpenSSL dependency, enabling pure Rust cross-compilation.

**Full release workflow**: [docs/development/RELEASE.md](/docs/development/RELEASE.md)

---

## Microstructure Features

10 intra-bar features + 16 inter-bar lookback features computed in Rust during bar construction.

**Intra-bar**: `ofi`, `vwap_close_deviation`, `kyle_lambda_proxy`, `trade_intensity`, `volume_per_trade`, `aggression_ratio`, `aggregation_density`, `turnover_imbalance`, `duration_us`, `price_impact`

**Inter-bar** (lookback window): `lookback_ofi`, `lookback_intensity`, `lookback_kyle_lambda`, `lookback_burstiness`, `lookback_hurst`, `lookback_permutation_entropy`, + 10 more

**Formulas and ranges**: [crates/CLAUDE.md](/crates/CLAUDE.md#microstructure-features-v70)

**Validation**: [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md#validation-framework-v70)

---

## ClickHouse Infrastructure

All open deviation bar data served from **bigblack** (remote GPU host). No local ClickHouse.

| Host        | Thresholds (dbps)   | ClickHouse | Total Bars |
| ----------- | ------------------- | ---------- | ---------- |
| bigblack    | 250, 500, 750, 1000 | Native     | 260M+      |
| littleblack | 100                 | Docker     | —          |

**Connection mode**: `OPENDEVIATIONBAR_MODE=remote` (set in `.mise.toml`). Preflight: `mise run db:ensure`.

**Cache operations**: [python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) (schema, population, dedup hardening)

**Distributed jobs**: [scripts/CLAUDE.md](/scripts/CLAUDE.md) (pueue, per-year parallelization, autoscaler)

---

## Monitoring & Gap Prevention

Automated monitoring to prevent silent data loss in the ClickHouse cache. Born from the Feb 2026 incident where 5.5 days of BTCUSDT@500 bars were lost due to `try_cache_write()` silently swallowing connection errors.

### Root Cause (Resolved)

`try_cache_write()` in `orchestration/open_deviation_bars_cache.py` was fire-and-forget — `populate_cache_resumable()` reused it even though ClickHouse IS the destination, not an optional cache. When SSH tunnel dropped, writes failed silently for days while the checkpoint advanced.

### Defense-in-Depth (Issue #158 Phase 7)

**Write integrity**: Four layers prevent silent data loss. **Overlap prevention**: Four layers prevent duplicate bars from concurrent writers.

| Layer                | Mechanism                                               | Details                                                             |
| -------------------- | ------------------------------------------------------- | ------------------------------------------------------------------- |
| **Fatal writes**     | `_fatal_cache_write()` in populate path                 | Raises on ClickHouse failure instead of warning                     |
| **Circuit breaker**  | `fatal_write_breaker` (3 failures → OPEN, 60s recovery) | Prevents cascade during ClickHouse outages                          |
| **T-1 guard**        | `populate_cache_resumable()` clamps `end_date`          | Prevents crash on unavailable Binance Vision data                   |
| **Gap detection**    | `scripts/detect_gaps.py` + heartbeat timer              | Every 5 min via systemd, alerts via @obdpy_bot                      |
| **L1 Redis lock**    | `cache_write_lock()` per (symbol, threshold)            | Prevents concurrent writes; graceful degradation if Redis down      |
| **L2 Schema dedup**  | ORDER BY `first_agg_trade_id` in ReplacingMergeTree     | Native dedup on merge — same trades from different writers collapse |
| **L3 Heartbeat**     | Stathera overlap count every 5 min                      | Already exists in `health_heartbeat.py`                             |
| **L4 Cleanup timer** | `stathera_cleanup.py --apply` hourly via systemd        | Automated sweep, Telegram alert if bars deleted                     |

### Telegram Notifications (@obdpy_bot)

| Event                      | Behavior                                                            |
| -------------------------- | ------------------------------------------------------------------- |
| All checks pass            | Ephemeral green heartbeat, auto-deletes after 15 min (~2-3 visible) |
| Gap or regression detected | **Persistent** red alert, LOUD notification, stays forever          |
| Cache write failure        | Circuit breaker alert via `opendeviationbar.notify.telegram`        |

**Config**: `OPENDEVIATIONBAR_TELEGRAM_TOKEN` in `.mise.local.toml` (local) or `deploy/opendeviationbar.local.env` (bigblack). Chat ID in `.mise.toml`.

**Heartbeat**: `scripts/health_heartbeat.py` via systemd timer (`opendeviationbar-heartbeat.timer`, every 5 min). Checks: ClickHouse, sidecar, recency-backfill, kintsugi, disk, circuit breaker.

**Full details**: [scripts/CLAUDE.md](/scripts/CLAUDE.md#gap-detection--monitoring)

---

## Common Errors

| Error                                          | Cause                  | Fix                                        |
| ---------------------------------------------- | ---------------------- | ------------------------------------------ |
| `OpenDeviationBarProcessor has no attribute X` | Outdated binding       | `maturin develop`                          |
| `Invalid threshold_decimal_bps`                | Wrong units            | Use 250 for 0.25%                          |
| `High < Low` assertion                         | Bad input data         | Check sorting                              |
| `target-cpu=native` cross-compile error        | RUSTFLAGS pollution    | Use `RUSTFLAGS=""` or `.cargo/config.toml` |
| OOM with `include_microstructure=True`         | Large date range       | MEM-011 adaptive chunk size                |
| Duplicate ticks in Parquet cache               | Pre-v12.8 bug          | `scripts/deduplicate_parquet_cache.py`     |
| `SymbolNotRegisteredError`                     | Symbol not in registry | Edit `symbols.toml`, `maturin develop`     |

**Memory guards**: [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) (MEM-001 through MEM-013)

---

## CLAUDE.md Network (Hub-and-Spoke)

This file is the **hub**. Each spoke CLAUDE.md is loaded automatically when working in that directory.

| Directory                              | CLAUDE.md                                                                                     | SSoT For                                                           |
| -------------------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
| `/`                                    | This file                                                                                     | Project overview, API, architecture, principles                    |
| `/crates/`                             | [crates/CLAUDE.md](/crates/CLAUDE.md)                                                         | Rust crates, microstructure formulas, algorithm details            |
| `/python/opendeviationbar/`            | [python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md)                       | Python API, caching, validation, symbol registry, memory guards    |
| `/python/opendeviationbar/clickhouse/` | [python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) | ClickHouse schema, dedup hardening, cache population               |
| `/python/opendeviationbar/config/`     | [python/opendeviationbar/config/CLAUDE.md](/python/opendeviationbar/config/CLAUDE.md)         | Configuration via pydantic-settings, env var priority              |
| `/python/opendeviationbar/plugins/`    | [python/opendeviationbar/plugins/CLAUDE.md](/python/opendeviationbar/plugins/CLAUDE.md)       | FeatureProvider protocol, entry-point discovery, plugin authoring  |
| `/scripts/`                            | [scripts/CLAUDE.md](/scripts/CLAUDE.md)                                                       | Pueue ops, per-year parallelism, gap detection, monitoring scripts |
| `/scripts/systemd/`                    | [scripts/systemd/CLAUDE.md](/scripts/systemd/CLAUDE.md)                                       | 5 services, 2 timers, dependency chain, local.env pattern          |

---

## Terminology

| Term                    | Definition                                                                                                                                                                                                                                                                                       |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **dbps**                | 1 dbps = 0.00001 = 0.001%. Example: 250 dbps = 0.25%. **All threshold values use dbps.**                                                                                                                                                                                                         |
| **Ouroboros**           | Cyclical reset boundary (year/month/week). Resets processor state for reproducibility and cache-friendly processing.                                                                                                                                                                             |
| **Orphan Bar**          | Incomplete bar emitted when processor resets at an ouroboros boundary. Marked `is_orphan=True`. Monthly mode: 12/year; yearly: 1/year. Filter for ML: `df[~df.get("is_orphan", False)]`. Synonyms: "orphaned bar", "boundary bar".                                                               |
| **Dynamic Ouroboros**   | Forex-specific: reset at first tick after weekend gap, auto-handling DST shifts.                                                                                                                                                                                                                 |
| **Stathera**            | (σταθερά) Trade-ID alignment paradigm (Issue #158). Invariant: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id` for all consecutive bars within same (symbol, threshold). Uses aggregate trade IDs already in ClickHouse as source of truth. Replaces all time-based gap detection. |
| **Stathera Anomaly**    | A violation of the Stathera invariant. `tid_delta > 0` = gap (missing aggTrade records). `tid_delta < 0` = overlap (bars share aggTrade ranges from different processing sources).                                                                                                               |
| **Kintsugi**            | Self-healing gap reconciliation (Issue #115). Discovers Stathera gaps (shards) via trade-ID continuity check and repairs using Ariadne `fromId` pagination + Ouroboros.                                                                                                                          |
| **Shard**               | A Stathera gap — consecutive bars with `tid_delta > 0` (missing aggTrade records). Classified P0 (staleness), P1 (recent <48h), P2 (historical >=48h). Identified by `anchor_last_tid` and `resume_first_tid`. Repaired by Kintsugi.                                                             |
| **Circuit Breaker**     | `fatal_write_breaker` in `open_deviation_bars_cache.py`. States: CLOSED (normal), OPEN (3 consecutive failures, blocks writes), HALF_OPEN (60s recovery probe). Prevents cascade failures during ClickHouse outages.                                                                             |
| **Ariadne**             | (Ἀριάδνη) `fromId` pagination for REST API gap fill. Thread of trade IDs through the labyrinth — uses `agg_trade_id` anchors to fetch exactly the missing trades. Zero-gap by construction.                                                                                                      |
| **Antaeus**             | (Ανταίος) Resilient WebSocket reconnection. Backoff resets after stable connection (≥60s); jitter via `backon` prevents thundering herd. Named for the giant who regained strength by touching the earth.                                                                                        |
| **Lethe**               | (Λήθη) Silent data loss detection. Alerts when ring buffer drops bars during backpressure. Named for the river of forgetfulness — ensures nothing is silently forgotten.                                                                                                                         |
| **Argus**               | (Ἄργος Πανόπτης) Universal liveness watchdog. Tracks per-symbol last-bar time; alerts when any symbol goes silent for ≥30 min regardless of aggregate trade flow. The hundred-eyed, all-seeing.                                                                                                  |
| **Charon**              | (Χάρων) Dead-letter ferry. Sidecar autonomously replays failed-write Parquet files on startup and every 5 min, without requiring kintsugi. Ferryman carrying lost bars back to ClickHouse.                                                                                                       |
| **StreamManager**       | Managed streaming engine wrapping `LiveBarEngine` with adaptive rate limiting (Issue #162), trade-ID continuity tracking seeded from ClickHouse, and shadow validation diagnostics. Drop-in replacement used by sidecar since v12.49.                                                            |
| **AdaptiveRateLimiter** | Token bucket rate limiter for Binance REST API (Issue #162). Syncs with `X-MBX-USED-WEIGHT-1m` server header, 2400 weight/min budget, 20% safety margin. Paces gap-fill REST calls in `fill_reconnection_gap()`.                                                                                 |
| **Shadow Validation**   | Diagnostic background loop in StreamManager. Every 5s, fetches latest trade via REST and compares with WS-tracked trade IDs. Skips when rate limiter utilization > 60%. Divergences logged, not auto-corrected.                                                                                  |

<!-- gitnexus:start -->

# GitNexus MCP

This project is indexed by GitNexus as **opendeviationbar-py** (7737 symbols, 20106 relationships, 300 execution flows).

## Always Start Here

1. **Read `gitnexus://repo/{name}/context`** — codebase overview + check index freshness
2. **Match your task to a skill below** and **read that skill file**
3. **Follow the skill's workflow and checklist**

> If step 1 warns the index is stale, run `npx gitnexus analyze` in the terminal first.

## Skills

| Task                                         | Read this skill file                                        |
| -------------------------------------------- | ----------------------------------------------------------- |
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md`       |
| Blast radius / "What breaks if I change X?"  | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
| Trace bugs / "Why is X failing?"             | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md`       |
| Rename / extract / split / refactor          | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md`     |
| Tools, resources, schema reference           | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md`           |
| Index, status, clean, wiki CLI commands      | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md`             |

<!-- gitnexus:end -->
