# llmdebug Static Codebase Review (Post-Fix)

## Scope
- Eval harness patch application and timeout handling (`evals/run_eval.py`, `evals/validate_cases.py`)
- Eval analyzer input loading and summary diagnostics (`evals/analyze_results.py`)
- Serialization redaction safety and public config surface (`src/llmdebug/config.py`, `src/llmdebug/serialize.py`, `src/llmdebug/capture.py`, `src/llmdebug/__init__.py`)
- Documentation and regression coverage (`README.md`, `tests/`)

## Findings Status

### Fixed in this pass
1. **P0 path traversal in patch apply** — fixed  
   - Evidence: `evals/run_eval.py:677`, `evals/run_eval.py:779`  
   - Change: `apply_diff` now validates all diff paths before invoking `patch`/`git apply` and rejects unsafe/escaping paths plus `/dev/null`.
2. **P1 uncaught pytest timeouts in eval/validator** — fixed  
   - Evidence: `evals/run_eval.py:591`, `evals/run_eval.py:624`, `evals/validate_cases.py:206`  
   - Change: `run_pytest` now returns structured timeout results (`returncode=124`, `timed_out`, `timeout_sec`), and validator reports explicit timeout failures.
3. **P1 analyzer crashes on malformed JSONL/input files** — fixed  
   - Evidence: `evals/analyze_results.py:66`, `evals/analyze_results.py:82`, `evals/analyze_results.py:277`  
   - Change: analyzer now streams line-by-line, tolerates malformed records by default, and adds `--strict-input` for fail-fast mode.
4. **P2 non-atomic relaxed fallback writes** — fixed  
   - Evidence: `evals/run_eval.py:797`  
   - Change: relaxed replacement fallback now computes all edits first and only writes after full validation, preventing partial state on failure.
5. **P2 redaction key-collision data loss** — fixed  
   - Evidence: `src/llmdebug/config.py:21`, `src/llmdebug/serialize.py:28`, `src/llmdebug/serialize.py:438`  
   - Change: default redaction now preserves dict keys (`redact_keys=False`) and only redacts values; key redaction is opt-in.
6. **P3 analyzer memory pressure from full-file reads** — fixed  
   - Evidence: `evals/analyze_results.py:87`  
   - Change: analyzer now iterates files as streams instead of `read_text().splitlines()`.

### Residual risks (not addressed in this pass)
1. **P2 runtime metric incompleteness in analyzer**  
   - Evidence: `evals/analyze_results.py:199`  
   - Risk: aggregated runtime still depends on available fields and may omit some patch-side overhead if not recorded uniformly.
2. **P2 shell execution risk in command patcher (by-design but sharp edge)**  
   - Evidence: `evals/patchers/command_patcher.py:26`  
   - Risk: `shell=True` allows arbitrary command execution via configured patch command; acceptable for local trusted workflows, risky otherwise.
3. **P3 pytest payload growth risk**  
   - Evidence: `src/llmdebug/pytest_plugin.py:127`  
   - Risk: large `longrepr`/captured output can still inflate snapshot payloads in pathological failures.

## Validation Performed
- Focused regression suite passed:
  - `tests/test_evals_apply_diff_fallback.py`
  - `tests/test_evals_analyze_grouping.py`
  - `tests/test_evals_run_pytest_timeout.py`
  - `tests/test_evals_validate_cases.py`
  - `tests/test_evals_force_patch_policy.py`
  - `tests/test_evals_case_loading.py`
  - `tests/test_evals_openai_patcher.py`
  - `tests/test_serialize.py`

## Recommended next remediation order
1. Normalize runtime accounting to a single end-to-end attempt metric in runner records and analyzer.
2. Decide policy for command patcher trust boundary (`shell=True` vs safer command invocation model).
3. Add optional truncation controls for pytest payload fields if token budget becomes a bottleneck.
