# llmdebug evals

This folder provides a local A/B harness to measure whether llmdebug snapshots
improve automated debugging outcomes across a deterministic synthetic suite.

- **A (`traceback_only`)**: pytest output only
- **B (`with_snapshot`)**: pytest output + `.llmdebug/latest.*`

Patcher backends:
- `command`: local command that returns a unified diff
- `openai`: OpenAI-compatible `/v1/chat/completions`
- `heuristic`: demo snapshot-aware patcher
- `null`: no-op patcher (smoke testing)

Artifacts are written to `evals/artifacts/` and results to `evals/results/*.jsonl`.

The suite is signal-focused:
- `snapshot_dependency=required`: bugs where runtime state should materially help
- `snapshot_dependency=helpful`: snapshots may help, but traceback may still suffice
- `snapshot_dependency=none`: traceback-control cases

## Case contract

Each `evals/cases/<case_id>/` must contain:
- `buggy.py` (or small module set),
- `test_buggy.py`,
- `case.json`,
- `oracle.patch` (required; used by validator).

`case.json` schema:

```json
{
  "description": "short summary",
  "category": "hidden_state|mutability_aliasing|async_temporal|cross_file_contract|data_shape_runtime|parse_value|path_import_env|traceback_controls|syntax_error|reference_error|logic_error|type_coercion",
  "difficulty": "easy|medium|hard",
  "snapshot_dependency": "required|helpful|none",
  "expected_exception": "IndexError",
  "tags": ["multifile", "stateful"],
  "python_args": ["-m", "pytest", "-q"],
  "timeout_sec": 30,
  "bug_source": "hand_crafted|debugbench|humanevalpack|condefects|mutmut",
  "contamination_risk": "none|low|medium|high",
  "expected_fix_lines": 3,
  "source_id": "debugbench_42",
  "difficulty_rationale": "optional explanation"
}
```

The last five fields (`bug_source` through `difficulty_rationale`) are optional
and used by imported cases for traceability.

### Categories

| Category | Description |
|----------|-------------|
| `hidden_state` | Bugs involving mutable state, caching, stale values |
| `mutability_aliasing` | Reference/aliasing issues, shared mutable objects |
| `async_temporal` | Async/await ordering, race conditions, cancellation |
| `cross_file_contract` | Multi-file contract violations, API misuse |
| `data_shape_runtime` | Array shape/dimension mismatches, wrong dtypes |
| `parse_value` | Parsing or value extraction bugs |
| `path_import_env` | Path/import/environment configuration issues |
| `traceback_controls` | Control cases where traceback alone suffices |
| `syntax_error` | Missing colons, wrong indentation, operator misuse |
| `reference_error` | Undefined variables, wrong scope, missing attributes |
| `logic_error` | Wrong comparisons, off-by-one, incorrect conditions |
| `type_coercion` | Type mismatch, wrong cast, implicit conversion bugs |

### Contamination risk

Imported cases carry a `contamination_risk` field indicating how likely the
bug-fix pair appears in LLM training data:

| Level | Description | Example sources |
|-------|-------------|-----------------|
| `none` | Freshly generated, impossible to memorize | hand-crafted, mutmut mutations |
| `low` | Post-training-cutoff or GPT-4-generated bugs | DebugBench, ConDefects |
| `medium` | Pre-cutoff but not widely published | Older benchmarks |
| `high` | Extremely well-known benchmark | HumanEvalPack |

## Validate cases (required before benchmarks)

```bash
uv run python -m evals.validate_cases --schema-only
```

Useful filters:

```bash
uv run python -m evals.validate_cases --category hidden_state --difficulty easy --snapshot-dependency required --max-cases 5
```

Full executable integrity validation (baseline fail + oracle pass) is available for
later runs:

```bash
uv run python -m evals.validate_cases
```

## Current case inventory

<!-- BEGIN AUTO-GENERATED: CASE_INVENTORY -->
_This section is auto-generated from `evals/cases` by `python -m evals.doc_sync --write`._

1664 validated cases across 12 categories and 3 sources:

| Source | Cases |
|--------|-------|
| ConDefects | 1267 |
| DebugBench | 328 |
| Hand-crafted | 69 |
| **Total** | **1664** |

Category distribution:

| Category | Count | Difficulty breakdown |
|----------|-------|---------------------|
| logic_error | 1204 | 589 easy, 302 medium, 313 hard |
| type_coercion | 294 | 198 easy, 47 medium, 49 hard |
| reference_error | 79 | 30 easy, 36 medium, 13 hard |
| async_temporal | 30 | 7 easy, 16 medium, 7 hard |
| syntax_error | 26 | 16 easy, 6 medium, 4 hard |
| data_shape_runtime | 6 | 3 easy, 2 medium, 1 hard |
| parse_value | 5 | 3 easy, 2 medium |
| cross_file_contract | 4 | 1 easy, 2 medium, 1 hard |
| hidden_state | 4 | 2 easy, 2 medium |
| mutability_aliasing | 4 | 2 easy, 1 medium, 1 hard |
| path_import_env | 4 | 3 easy, 1 medium |
| traceback_controls | 4 | 4 easy |
<!-- END AUTO-GENERATED: CASE_INVENTORY -->

## Import external cases

Install the `evals` extra for HuggingFace dataset access:

```bash
pip install llmdebug[evals]
# or: pip install datasets>=2.0
```

### DebugBench (contamination: low)

Imports Python cases from `Rtian/DebugBench` on HuggingFace. Filters by code length,
fix size, determinism, and stdlib-only imports. Generates tests from LeetCode examples.
~34% of imported cases survive full execution validation.

```bash
# Full import (unlimited cases, auto-validate)
uv run python -m evals.importers.cli debugbench --validate

# Preview first 5 cases without writing
uv run python -m evals.importers.cli debugbench --max-cases 5 --dry-run
```

### HumanEvalPack (~20 cases, contamination: high)

```bash
uv run python -m evals.importers.cli humanevalpack --max-cases 20 --validate
```

### ConDefects (contamination: low)

Requires cloning the ConDefects repo and downloading test data separately.

```bash
git clone https://github.com/appmlk/ConDefects.git
# Download Test.zip from OneDrive and extract into ConDefects/
uv run python -m evals.importers.cli condefects --repo-path ./ConDefects --validate
```

### mutmut mutations (contamination: none)

Generates zero-contamination cases by mutating llmdebug source code itself.
Only keeps killed mutants (where tests actually detect the bug).

```bash
pip install mutmut
uv run python -m evals.importers.cli mutmut --target src/llmdebug/capture.py --max-cases 10
uv run python -m evals.importers.cli mutmut --target src/llmdebug/serialize.py --max-cases 10
```

## Benchmark runs

Quick sanity (small subset):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --openai-response-mode edits \
  --no-files-context \
  --k 1 \
  --max-cases 5
```

Full suite (all discovered cases in `evals/cases`):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --openai-response-mode edits \
  --no-files-context \
  --k 1
```

Filtered run (category/difficulty/snapshot dependency/bug source):

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --category async_temporal \
  --difficulty hard \
  --snapshot-dependency required \
  --k 3

# Filter by bug source
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --bug-source hand_crafted \
  --k 1
```

`--openai-response-mode edits` is recommended for local reasoning models.
`--no-files-context` is useful when prompts are too long.
Runner-level forced patch attempts are enabled by default to reduce `no_patch_proposed`
and `patch_apply_failed` records; disable with `--no-force-patch-attempt`.
For OpenAI patcher responses specifically, `--openai-force-patch-attempt` can further
force a minimal patch diff when model output is unparsable.

### Reproducibility controls

Use deterministic mode when you need reproducible patch-generation runs:

```bash
uv run python -m evals.run_eval \
  --patcher openai \
  --openai-base-url http://127.0.0.1:1234 \
  --openai-model deepseek/deepseek-r1-0528-qwen3-8b \
  --deterministic \
  --seed 12345 \
  --record-prompt hash \
  --record-evidence hash \
  --record-deps minimal
```

Key flags:
- `--deterministic --seed <int>`: enforce deterministic mode; fails fast if OpenAI-compatible backend does not acknowledge seed support.
- `--record-prompt full|hash|none`: controls prompt persistence in `attempt_metadata`.
- `--record-evidence full|hash|none`: controls evidence persistence in `attempt_metadata`.
- `--record-env KEY` (repeatable): allowlist environment keys into `run_metadata`.
- `--record-deps none|minimal|runtime`: dependency version capture scope.

## Analyze and report uplift

```bash
uv run python -m evals.analyze_results evals/results/*.jsonl --out evals/results/summary.json
```

`evals.analyze_results` now enforces strict schema validation (`schema_version=2`) for
records. Older JSONL outputs without required metadata fields will be rejected and should
be regenerated with the current harness.

The analyzer reports:
- overall metrics,
- per-category metrics,
- per-difficulty metrics,
- per-snapshot-dependency metrics,
- per-bug-source metrics,
- per-contamination-risk metrics,
- explicit `uplift_success_rate = with_snapshot - traceback_only`.

### Statistical analysis

Add `--stats` for rigorous significance testing:

```bash
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --out evals/results/summary.json
```

The `--stats` flag adds:
- **McNemar's test**: Paired binary significance test (exact binomial for <25 discordant pairs, chi-squared otherwise).
- **Cohen's h**: Effect size for proportions (small >= 0.2, medium >= 0.5, large >= 0.8).
- **Bootstrap CI**: 95% confidence interval for uplift via paired bootstrap (10k resamples).
- **Holm-Bonferroni**: Multiple comparison correction for per-category p-values.
- **Power report**: Post-hoc power assessment to gauge result trustworthiness.

For multi-run (k>1) analysis, use `--aggregation`:

```bash
# Majority vote: case succeeds if >50% of runs succeed
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --aggregation majority_vote

# Any success: case succeeds if any run succeeds
uv run python -m evals.analyze_results evals/results/*.jsonl --stats --aggregation any_success
```

### Sample size guidance

| Cases | Discordant pairs (est.) | Power (uplift=0.2) | Recommendation |
|-------|------------------------|---------------------|----------------|
| 30 | ~10 | ~0.30 | Directional signal only |
| 60 | ~20 | ~0.55 | Moderate evidence |
| 100 | ~35 | ~0.75 | Near-publication quality |
| 150+ | ~50+ | ~0.90+ | Publication-grade |

Interpretation guidance:
- positive uplift in `required` buckets suggests llmdebug snapshots provide useful signal,
- near-zero uplift in `none` control buckets is expected and acts as a sanity check,
- positive uplift in a category suggests snapshots help that bug class,
- near-zero uplift suggests no clear advantage yet,
- negative uplift suggests prompt or patch quality regressions for that bucket.
