# Scripts & Operations

**Parent**: [/CLAUDE.md](/CLAUDE.md) | **Skills**: `Skill(devops-tools:pueue-job-orchestration)` | `Skill(opendeviationbar-job-safety)`

Operational scripts for cache population, validation, and distributed job management on remote GPU hosts.

---

## Quick Reference

| Task                       | Command                                           | Details                                               |
| -------------------------- | ------------------------------------------------- | ----------------------------------------------------- |
| Queue all cache population | `mise run cache:populate-all`                     | [Pueue Population](#pueue-population)                 |
| Check pueue status         | `mise run cache:populate-status`                  | [Monitoring](#monitoring)                             |
| Per-year parallelization   | `--start-date` / `--end-date`                     | [Per-Year Parallelization](#per-year-parallelization) |
| Autoscale parallelism      | `mise run cache:autoscale-loop`                   | [Autoscaler](#autoscaler)                             |
| Check cache status         | `mise run cache:status`                           | ClickHouse bar counts                                 |
| Detect volume overflow     | `mise run cache:detect-overflow`                  | Issue #88                                             |
| Validate dedup hardening   | `mise run cache:validate-dedup`                   | Issue #90                                             |
| Validate microstructure    | `uv run python scripts/validate_microstructure_*` | [Validation Scripts](#validation-scripts)             |
| Run streaming sidecar      | `mise run streaming:start`                        | v12.20+, real-time open deviation bars                |
| Kintsugi (single pass)     | `mise run kintsugi`                               | [Kintsugi](#kintsugi)                                 |
| Kintsugi (daemon)          | `mise run kintsugi:daemon`                        | Long-running self-healer                              |
| Kintsugi (dry-run)         | `mise run kintsugi:check`                         | Discover shards without repairing                     |

---

## Pueue Population

**State-of-the-art approach** for long-running cache population on remote hosts. Pueue daemon survives SSH disconnects, crashes, and reboots.

### Architecture

```
mise run cache:populate-all
  └─→ pueue-populate.sh
        └─→ pueue add (per symbol × threshold)
              └─→ [systemd-run --scope -p MemoryMax=XG]  (Linux only)
                    └─→ uv run python populate_full_cache.py
                          └─→ populate_cache_resumable()   (day-by-day, checkpointed)
```

### Key Files

| Script                      | Purpose                                         |
| --------------------------- | ----------------------------------------------- |
| `pueue-populate.sh`         | Pueue job orchestrator (groups, phases, guards) |
| `populate_full_cache.py`    | Single-job entry point (Python)                 |
| `pueue-autoscaler.sh`       | Dynamic parallelism tuning (CPU/memory aware)   |
| `detect_volume_overflow.py` | Post-population integrity check (Issue #88)     |
| `streaming_sidecar.py`      | Streaming sidecar CLI entry point (v12.20+)     |

### Phases

| Phase | Threshold (dbps) | Pueue Group | Parallel | Memory/Job (BTC/ETH) | Memory/Job (Altcoins) |
| ----- | ---------------- | ----------- | -------- | -------------------- | --------------------- |
| 1     | 1000             | p1          | 6        | ~6 GB (cap 8G)       | ~3 GB (cap 4G)        |
| 2     | 250              | p2          | 2        | ~2 GB (cap 8G)       | ~1 GB (cap 4G)        |
| 3     | 500, 750         | p3          | 3        | ~4 GB (cap 6G)       | ~2 GB (cap 4G)        |

> **Note**: Memory scales with bars-per-day × features. BTC/ETH generate 2-4x more bars than altcoins, requiring proportionally more memory. All measurements with `--include-microstructure` (26 features). See `pueue-populate.sh` for SSoT.

### Setup (One-Time on Remote Host)

```bash
# Install pueue
~/.local/bin/pueued -d                      # Start daemon
./scripts/pueue-populate.sh setup           # Configure groups

# Or via mise
mise run cache:populate-setup
```

### Resource Guards

On Linux with cgroups v2, each job runs inside a `systemd-run` scope with per-threshold memory caps and `MemorySwapMax=0` (prevents swap escape). Bypass with `OPENDEVIATIONBAR_NO_CGROUP=1`.

---

## Per-Year Parallelization (Default Strategy)

**This is the default approach for all multi-year cache population.** Never queue a monolithic multi-year job. Ouroboros resets processor state at year boundaries, making each year an independent processing unit. Per-year splits provide massive speedup on multi-core hosts (22 days → 3-4 days for DOGEUSDT@500).

### Why It's Safe (Three Isolation Layers)

| Layer                 | Why No Conflicts                                                                          |
| --------------------- | ----------------------------------------------------------------------------------------- |
| **Checkpoint files**  | `_get_checkpoint_path()` uses `{symbol}_{threshold}_{start}_{end}.json` — unique per year |
| **ClickHouse writes** | INSERT is append-only; `OPTIMIZE TABLE FINAL` deduplicates afterward                      |
| **Tick data**         | Read-only Parquet files; no write contention                                              |

### Usage

```bash
# populate_full_cache.py supports --start-date and --end-date overrides
uv run python scripts/populate_full_cache.py \
    --symbol SHIBUSDT --threshold 250 --include-microstructure \
    --start-date 2021-05-10 --end-date 2021-12-31

# Queue per-year jobs via pueue
pueue group add yearly --parallel 5
for year in 2021 2022 2023 2024 2025; do
    pueue add --group yearly --label "SHIB@250:${year}" -- \
        uv run python scripts/populate_full_cache.py \
        --symbol SHIBUSDT --threshold 250 --include-microstructure \
        --start-date "${year}-01-01" --end-date "${year}-12-31"
done
```

### Critical Rules

1. **Working directory**: Always `cd ~/opendeviationbar-py &&` before `pueue add` — SSH cwd defaults to `$HOME`, causing instant job failure with `No such file or directory`.
2. **No `--force-refresh` on per-year jobs** when other year-jobs are running — `force_refresh` deletes cached bars for the specified date range, but if the original full-range job already ran `force_refresh`, the cache is already empty.
3. **First year uses symbol's `effective_start`** from `symbols.toml`, not `01-01`.
4. **Last year uses `probe_latest_available_date()`** as end date.
5. **Chain `OPTIMIZE TABLE FINAL`** after all year-jobs complete via `pueue add --after`.
6. **Memory budget**: Each job peaks at ~1-6 GB depending on threshold and symbol (BTC/ETH use 2-4x more than altcoins with `--include-microstructure`). With 61 GB total, 6-8 concurrent year-jobs are safe for altcoins, 3-4 for BTC/ETH at heavy thresholds.

### Anti-Patterns (Hard Lessons)

| Anti-Pattern                       | What Happened                                                                                  | Fix                                              |
| ---------------------------------- | ---------------------------------------------------------------------------------------------- | ------------------------------------------------ |
| Monolithic multi-year job          | DOGEUSDT@500 estimated 22 days single-threaded                                                 | Split into per-year pueue jobs (22d → 3-4d)      |
| Wrong cwd in remote pueue          | `ssh host "pueue add ..."` used `$HOME` as cwd                                                 | `cd ~/opendeviationbar-py && pueue add ...`      |
| OPTIMIZE TABLE timeout             | Synchronous OPTIMIZE timed out at 300s under load                                              | Made OPTIMIZE non-fatal (Issue #90)              |
| Cascade dependency failure         | OPTIMIZE TABLE chained `--after` populate jobs; when any populate failed, OPTIMIZE was skipped | Run OPTIMIZE manually after retrying failed jobs |
| Per-job env override bypasses SSoT | `env OPENDEVIATIONBAR_CRYPTO_MIN_THRESHOLD=250` in each pueue job                              | Set in `.mise.toml [env]` instead (one place)    |

### When to Use Per-Year vs Sequential

| Scenario                                | Approach                 |
| --------------------------------------- | ------------------------ |
| High-volume symbol (SHIBUSDT, DOGEUSDT) | Per-year (5+ cores idle) |
| Low-volume symbol (NEARUSDT, ATOMUSDT)  | Sequential (fast enough) |
| Single threshold, long backfill         | Per-year                 |
| Multiple thresholds, same symbol        | Sequential per threshold |

### Performance Context

SHIBUSDT@250 on 2021-05-10 (listing pump day): **3.1M trades → 128K bars**. Single-threaded intra-bar feature computation (Hurst DFA, Permutation Entropy) takes 2-4 hours for this one day. Per-year parallelization moves later years forward while 2021 grinds through the hard days.

---

## Autoscaler

Pueue has no resource awareness. The autoscaler complements it with dynamic parallelism tuning.

```bash
mise run cache:autoscale         # Dry-run (shows what would change)
mise run cache:autoscale-apply   # Apply changes
mise run cache:autoscale-loop    # Continuous (60s interval)
```

**Scaling thresholds**:

```
CPU < 40% AND MEM < 60%  →  SCALE UP (+1 per group)
CPU > 80% OR  MEM > 80%  →  SCALE DOWN (-1 per group)
Otherwise                 →  HOLD
```

---

## Monitoring

```bash
# Pueue status
pueue status                          # All groups
pueue status --group p2               # Specific group
pueue follow <id>                     # Watch live output
pueue log <id> --lines 20            # Recent output

# Cache status
mise run cache:status                 # ClickHouse bar counts
python scripts/cache_status.py        # Direct

# System resources (bigblack)
ssh bigblack 'uptime && free -h'
```

---

## Gap Detection & Monitoring

Automated gap detection to prevent silent data loss in ClickHouse. Born from the Feb 2026 BTCUSDT@500 incident (5.5-day gap caused by `try_cache_write()` silently swallowing SSH tunnel failures).

### Scripts

| Script                           | Purpose                                            | Exit Code              |
| -------------------------------- | -------------------------------------------------- | ---------------------- |
| `detect_gaps.py`                 | Find temporal/price gaps + freshness in ClickHouse | 0=clean, 1=gaps, 2=err |
| `verify_checkpoint_integrity.py` | Cross-validate checkpoint files vs ClickHouse      | 0=clean, 1=mismatch    |
| `setup_daily_gap_check.sh`       | Install daily pueue monitoring job on bigblack     | —                      |

### Usage

```bash
# Direct script usage (no mise task exists)
python scripts/detect_gaps.py --help
python scripts/detect_gaps.py --symbol BTCUSDT --threshold 500
python scripts/detect_gaps.py --recent-days 7 --json

# Freshness check (flag stale data >72 hours since last bar)
python scripts/detect_gaps.py --recent-days 7 --max-stale-hours 72

# Checkpoint integrity
python scripts/verify_checkpoint_integrity.py --quiet
```

### Freshness Detection

The `--max-stale-hours N` flag computes `now - latest_bar` for each (symbol, threshold) pair. If any pair hasn't produced a bar in N hours, it's flagged as stale and triggers exit code 1. The heartbeat uses `--max-stale-hours 72` (3 days).

This catches scenarios where ALL thresholds for a symbol stop simultaneously (e.g., backfill_watcher not running) — something inter-bar gap detection alone misses because it only checks gaps between consecutive bars within existing data.

### Telegram Alerts (@obdpy_bot)

Gap detection integrates with `opendeviationbar.notify.telegram`:

- **Gaps found** → persistent alert (stays forever, LOUD notification)
- **All clear** → no message (or ephemeral heartbeat via launchd)
- **`--no-notify`** flag suppresses Telegram (for dry runs)

### Heartbeat (systemd timer — bigblack only)

`scripts/health_heartbeat.py` runs every 5 minutes via `opendeviationbar-heartbeat.timer`. Self-contained (no package imports for Telegram API — avoids version mismatch with installed PyPI package).

| Check            | Method                              | Threshold   |
| ---------------- | ----------------------------------- | ----------- |
| ClickHouse       | `SELECT 1` via `clickhouse_connect` | Reachable   |
| sidecar          | `systemctl --user is-active`        | Active      |
| recency-backfill | `systemctl --user is-active`        | Active      |
| kintsugi         | `systemctl --user is-active`        | Active      |
| Disk             | `shutil.disk_usage("/")`            | >10 GB free |
| Circuit breaker  | `get_fatal_write_breaker().state`   | Not OPEN    |

**Healthy**: Ephemeral message, tracked in `~/.cache/opendeviationbar/heartbeat/pending_messages.json`, auto-deleted after 15 min (~2-3 visible at any time).

**Unhealthy**: LOUD persistent message, never auto-deleted.

**Systemd files**: `scripts/systemd/opendeviationbar-heartbeat.{service,timer}` | [systemd CLAUDE.md](/scripts/systemd/CLAUDE.md)

### Gap Detection Query (SSoT)

Uses ClickHouse `lagInFrame()` window function to compute inter-bar time deltas within each `(symbol, threshold)` partition. Gaps exceeding `--min-gap-hours` (default 6h) are flagged. Note: `neighbor()` is deprecated in ClickHouse 24.x+.

---

## Kintsugi

Self-healing gap reconciliation (Issue #115). Discovers interior gaps ("shards") in the ClickHouse cache and repairs them autonomously using Ariadne (fromId pagination) + Ouroboros (boundary-aware splits).

| Script/File        | Purpose                                |
| ------------------ | -------------------------------------- |
| `kintsugi.py`      | CLI entry point + daemon mode          |
| `kintsugi.toml`    | mise task definitions (`.mise/tasks/`) |
| `kintsugi.service` | systemd unit (`scripts/systemd/`)      |

### Core engine: `python/opendeviationbar/kintsugi.py`

- `discover_shards()` — Stathera trade-ID audit via direct ClickHouse query (Issue #158)
- `repair_shard()` — P0/P1 via `backfill_recent(overlap_strategy="replace")`, P2 via `populate_cache_resumable()`
- `verify_repair()` — Stathera verification via ClickHouse (checks trade-ID gap is healed)
- `kintsugi_pass()` — single discover-repair-verify cycle
- `kintsugi_daemon()` — long-running adaptive polling (30 min clean, 15 min active)

### Safeguards

- Max 3 repair attempts per shard per 24h (rate-limited via `kintsugi_log`)
- P2 jobs capped by `--max-p2-jobs` (default 3, via `_DEFAULT_MAX_P2_JOBS`)
- Sidecar dead-letter replay on daemon startup
- Telegram alerts on repair failures

---

## Stathera (σταθερά)

Trade-ID alignment paradigm (Issue #158). Replaces all time-based gap detection with aggregate trade-ID continuity checking. The invariant: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id` for all consecutive bars within the same (symbol, threshold) pair.

### Core Concepts

| Concept                | Definition                                                                                              |
| ---------------------- | ------------------------------------------------------------------------------------------------------- |
| **Stathera Invariant** | Trade-ID continuity: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id` for consecutive bars |
| **tid_delta**          | `first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) - 1` per bar. >0 = gap, =0 = clean, <0 = overlap |
| **Stathera Gap**       | A violation where `tid_delta > 0` (missing aggTrade records between consecutive bars)                   |
| **Stathera Overlap**   | A violation where `tid_delta < 0` (bars from different sources share trade-ID ranges)                   |

### Scripts

| Script                               | Purpose                                                                                           |
| ------------------------------------ | ------------------------------------------------------------------------------------------------- |
| `detect_gaps.py --stathera`          | **Default**: Trade-ID audit for all (symbol, threshold) pairs. Exit code 1 if gaps/overlaps found |
| `detect_gaps.py --legacy-time-based` | Legacy time-based detection (deprecated, moved behind flag)                                       |
| `stathera_cleanup.py`                | Identify overlapping bars, select canonical bars, delete redundant ones. Dry-run by default       |
| `stathera_cleanup.py --apply`        | Execute cleanup and re-audit to verify overlaps dropped to 0                                      |

### How Stathera Audit Works

1. **Audit Query** (`detect_gaps.py --stathera`):
   - Window function ordered by `(first_agg_trade_id, close_time_ms)` (critical: fixes overlap detection)
   - Computes `tid_delta = first_agg_trade_id - lagInFrame(last_agg_trade_id, 1) - 1`
   - Classifies: gap (>0), clean (=0), overlap (<0), legacy (first_agg_trade_id=0)

2. **Overlap Cleanup** (`stathera_cleanup.py`):
   - Identifies chains of overlapping bars (consecutive bars sharing trade IDs)
   - Selects canonical bars via greedy non-overlapping cover (prefers recent `computed_at`)
   - Deletes redundant bars via `ALTER TABLE DELETE`
   - Waits for mutations to complete, re-audits to confirm 0 overlaps

3. **Source Tagging** (Phase 2):
   - Each bar's `opendeviationbar_version` encoded as `{version}+{source}`:
     - `+sidecar` — single writer (cannot overlap itself)
     - `+recency` — REST backfill (skip if sidecar already wrote)
     - `+kintsugi` — repair processing (authoritative, can replace)
     - `+populate` — cache population (warn on overlaps, investigate)

### Integration with Kintsugi

- `discover_shards_stathera()` — direct ClickHouse Stathera audit, no subprocess
- Per-gap trade-ID anchors (`anchor_last_tid`, `resume_first_tid`) replace global high-water-mark
- Repair routing: P0/P1 via `backfill_recent()`, P2 via `populate_cache_resumable()`

### Verification

```bash
# Local
python scripts/detect_gaps.py --stathera --json   # Audit all pairs
python scripts/stathera_cleanup.py                # Dry-run cleanup

# Bigblack
python scripts/detect_gaps.py --symbol BTCUSDT --threshold 250 --stathera
python scripts/stathera_cleanup.py --apply        # Execute cleanup
```

---

## Streaming Reliability Concepts

The sidecar uses a layered self-healing architecture with named concepts from Greek mythology:

| Concept      | Layer      | Purpose                                                                                      |
| ------------ | ---------- | -------------------------------------------------------------------------------------------- |
| **Antaeus**  | Rust (WS)  | Resilient reconnection: `backon` exponential backoff with jitter, resets after stable (≥60s) |
| **Lethe**    | Python     | Ring buffer overflow alerting: detects `dropped_bars` metric increases, Telegram alerts      |
| **Argus**    | Python     | Per-symbol liveness: tracks last bar time per symbol, alerts if silent ≥30 min               |
| **Charon**   | Python     | Dead-letter ferry: auto-replays failed Parquet writes on startup + every 5 min               |
| **Ariadne**  | Python     | `fromId` REST pagination for gap-fill: trade-ID anchored, zero-gap by construction           |
| **Kintsugi** | Python     | Self-healing gap reconciliation via Stathera trade-ID audit + Ariadne repair                 |
| **Stathera** | ClickHouse | Trade-ID alignment invariant: `bar[i].last_agg_trade_id + 1 == bar[i+1].first_agg_trade_id`  |

### Configuration (Environment Variables)

| Variable                                 | Default | Purpose                         |
| ---------------------------------------- | ------- | ------------------------------- |
| `OPENDEVIATIONBAR_LETHE_ALERT_THRESHOLD` | `1`     | Alert on first ring buffer drop |
| `OPENDEVIATIONBAR_ARGUS_TIMEOUT_S`       | `1800`  | Per-symbol silence timeout (s)  |
| `OPENDEVIATIONBAR_CHARON_INTERVAL_S`     | `300`   | Dead-letter replay interval (s) |

---

## Validation Scripts

Portable scripts for GPU workstations without full dev environment:

| Script                                 | Purpose                             |
| -------------------------------------- | ----------------------------------- |
| `validate_n_open_deviation_bars.py`    | Count-bounded API validation        |
| `validate_microstructure_features.py`  | v7.0 feature validation             |
| `validate_microstructure_roundtrip.py` | Compute → store → read → verify     |
| `validate_backfill_preflight.py`       | Pre-population checks               |
| `validate_clickhouse.py`               | ClickHouse connectivity             |
| `validate_memory_efficiency.py`        | Memory usage during processing      |
| `deduplicate_parquet_cache.py`         | Fix pre-v12.8 duplicate ticks (#78) |
| `detect_volume_overflow.py`            | Find negative volumes (#88)         |
| `validate_dedup_hardening.py`          | Issue #90 dedup layer validation    |

---

## Related

- [/CLAUDE.md](/CLAUDE.md) - Project hub
- [/crates/CLAUDE.md](/crates/CLAUDE.md) - Rust crate details, microstructure features
- [/python/opendeviationbar/CLAUDE.md](/python/opendeviationbar/CLAUDE.md) - Python API, caching, validation
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) - ClickHouse cache layer
- [/.mise/tasks/cache.toml](/.mise/tasks/cache.toml) - mise task definitions
