# llmdebug Static Codebase Review (Round 2)

## Scope
- Follow-up review after full executable case validation.
- Focused on benchmark-suite consistency and validator behavior in `evals/`.

## Findings

### P2: Case ID suffix and difficulty metadata diverge
- **Evidence**: `evals/cases/parse_json_numeric_key_medium/case.json:4`, `evals/cases/shape_batch_axis_flip_medium/case.json:4`
- **Risk**: Case IDs imply `medium` but metadata marks them as `easy`, which can mislead filtering and difficulty-level uplift interpretation.
- **Recommendation**: Align IDs and metadata (either rename case IDs or set `difficulty` to `medium` and rebalance the target split explicitly).

### P2: Validator reports misleading failures when pytest is missing
- **Evidence**: `evals/validate_cases.py:134`, `evals/validate_cases.py:145`
- **Risk**: Environment errors (e.g., `No module named pytest`) are surfaced as `"expected exception not found"` for every case, masking root cause and wasting debugging time.
- **Recommendation**: Add a preflight check for pytest availability or detect/import-error signatures and return a dedicated environment/setup failure message.

### P2: Exception-type check remains substring-based
- **Evidence**: `evals/validate_cases.py:145`
- **Risk**: False positives/negatives remain possible when exception text formatting changes (or when class names are absent from output format).
- **Recommendation**: Parse exception class from pytest failure output deterministically and compare normalized exception names.

### P3: Contract checker still enforces fixed `buggy.py` filename
- **Evidence**: `evals/validate_cases.py:16`, `evals/validate_cases.py:107`
- **Risk**: Case authoring is constrained for multifile setups despite contract wording permitting small module sets.
- **Recommendation**: Require `test_buggy.py`, `case.json`, `oracle.patch`, and at least one non-test Python source file rather than a fixed `buggy.py`.

## Summary
- Full executable validation passes, but benchmark metadata consistency and validator diagnostics still have quality gaps that can distort analysis workflow.
- Recommended next change set is small and isolated to `evals/validate_cases.py` plus two case metadata files (or matching case renames).
