# systemd Services & Timers

**Parent**: [/scripts/CLAUDE.md](/scripts/CLAUDE.md) | **Hub**: [/CLAUDE.md](/CLAUDE.md)

All services run as `--user` units on bigblack. Managed via `mise run deploy:bigblack`.

---

## Services (6 units)

| Service                             | Type            | Purpose                              | Memory (High/Max) |
| ----------------------------------- | --------------- | ------------------------------------ | ----------------- |
| `opendeviationbar-sidecar`          | simple (daemon) | Real-time streaming via WebSocket    | 2G / 2.5G         |
| `opendeviationbar-recency-backfill` | simple (daemon) | Bridge Vision lag via REST API       | 1G / 1.5G         |
| `opendeviationbar-kintsugi`         | notify (daemon) | Self-healing gap repair              | 8G / 10G          |
| `opendeviationbar-daily-gap-check`  | oneshot (timer) | Exhaustive daily gap detection       | 2G                |
| `opendeviationbar-heartbeat`        | oneshot (timer) | Ephemeral health check to Telegram   | 256M              |
| `opendeviationbar-stathera-cleanup` | oneshot (timer) | Hourly Stathera overlap cleanup (L4) | 1G                |

## Timers (3)

| Timer                                     | Interval        | Service          |
| ----------------------------------------- | --------------- | ---------------- |
| `opendeviationbar-daily-gap-check.timer`  | Daily 06:00 UTC | daily-gap-check  |
| `opendeviationbar-heartbeat.timer`        | Every 5 min     | heartbeat        |
| `opendeviationbar-stathera-cleanup.timer` | Every hour      | stathera-cleanup |

---

## Dependency Chain

```
sidecar → recency-backfill → kintsugi → daily-gap-check
```

Each service uses `After=` to ensure predecessor is started first. The heartbeat timer is independent (runs regardless of daemon state — that's what it monitors).

---

## Environment Files

All services load TWO env files:

1. `deploy/opendeviationbar.env` — git-tracked, non-secret config
2. `deploy/opendeviationbar.local.env` — **NOT** git-tracked, secrets (Telegram token, etc.)

The `-` prefix in `EnvironmentFile=-` means "ignore if missing" (used for `local.env` so services start even without secrets).

---

## mimalloc

Long-running services (sidecar, recency-backfill, kintsugi) use:

```ini
Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libmimalloc.so.2
Environment=MIMALLOC_PURGE_DELAY=1000
```

Prevents glibc ptmalloc2 RSS creep by returning freed memory to the OS. Without this, long-running Python+Rust processes accumulate unreclaimable RSS over days.

---

## OOM Protection

All long-running services use `MemorySwapMax=0` (no swap escape) and `ManagedOOMMemoryPressure=kill` for graceful systemd-oomd degradation via PSI instead of hard kernel OOM kill.

Priority order via `OOMScoreAdjust`:

1. sidecar (-500) — highest priority, latency-critical
2. recency-backfill (-400) — important but not latency-critical
3. kintsugi (-200) — background repair

---

## Deploy

`mise run deploy:bigblack` syncs service files, enables timers, and restarts all services.

---

## Related

- [/scripts/CLAUDE.md](/scripts/CLAUDE.md) - Operations scripts, gap detection, kintsugi
- [/python/opendeviationbar/clickhouse/CLAUDE.md](/python/opendeviationbar/clickhouse/CLAUDE.md) - ClickHouse cache, circuit breaker
- [/CLAUDE.md](/CLAUDE.md) - Project hub
