# llmdebug Research & Improvement Roadmap

> Deep research synthesis from 10 parallel investigations covering: competitive landscape,
> academic papers (2024-2026), MCP protocol, pytest ecosystem, token optimization,
> differential debugging, production patterns, Jupyter/CLI integration, eval frameworks,
> and developer ecosystem adoption.
>
> **Date:** 2026-02-07
> **Methodology:** Web research across academic databases, GitHub, PyPI, and industry blogs

---

## Table of Contents

- [Executive Summary](#executive-summary)
- [1. Competitive Landscape](#1-competitive-landscape)
- [2. Academic Research on LLM Debugging](#2-academic-research-on-llm-debugging)
- [3. Token Optimization Strategies](#3-token-optimization-strategies)
- [4. MCP Server Integration](#4-mcp-server-integration)
- [5. Pytest Plugin Enhancements](#5-pytest-plugin-enhancements)
- [6. Differential Debugging & Snapshot Diffing](#6-differential-debugging--snapshot-diffing)
- [7. Production Crash Capture](#7-production-crash-capture)
- [8. Jupyter & CLI Integration](#8-jupyter--cli-integration)
- [9. Eval Framework Improvements](#9-eval-framework-improvements)
- [10. Developer Ecosystem & Adoption](#10-developer-ecosystem--adoption)
- [Prioritized Improvement Roadmap](#prioritized-improvement-roadmap)
- [Packaging Strategy](#packaging-strategy)
- [Research-Backed Claims](#research-backed-claims)
- [Sources](#sources)

---

## Executive Summary

llmdebug occupies a unique position in the debugging tool landscape: it captures **structured
crash snapshots optimized for LLM consumption** at test time. Most competitors focus on either
real-time IDE assistance (Cursor, Copilot) or production monitoring (Sentry, Datadog). No other
tool combines zero-config pytest integration, ML tensor introspection, compact token-efficient
output, and an A/B evaluation harness for measuring snapshot effectiveness.

**Three strategic insights from this research:**

1. **MCP is the unlock** - The Model Context Protocol is becoming the universal bridge between
   AI coding tools and data sources. An MCP server would make llmdebug instantly available in
   Claude Code, Cursor, VS Code Copilot, and any future MCP-capable tool.

2. **Less context > more context** - Academic research consistently shows focused, structured
   debugging context outperforms large dumps. llmdebug's crash-site locals + array metadata +
   compact format approach is directly validated by SWE-Pruner and LDB research.

3. **The evals are the moat** - The A/B evaluation harness (traceback-only vs. with-snapshot)
   is rare in the debugging tool space. Expanding it with statistical rigor creates publishable,
   citeable evidence that no competitor can match.

### Status Update (2026-02-10)

This document was originally written as a forward-looking research plan. Since publication,
several items from the early roadmap have shipped.

**Shipped capabilities (now available):**
- MCP server for IDE integrations
- CLI snapshot workflows
- Snapshot diffing
- Jupyter/IPython integration
- Production exception hooks and middleware capture
- Hypothesis generation
- Redaction policy profiles (`dev`/`ci`/`prod`) with explicit override controls
- Deterministic eval reproducibility controls and strict eval schema validation
- RCA prompt contract and gated RCA state workflows (MCP-integrated)

**Open gaps (still roadmap priorities):**
- Historical/indexed snapshot search
- Git bisect + deeper git-aware debugging workflows
- Shareable report/dashboard experience

---

## 1. Competitive Landscape

### Direct Competitors

| Tool | Focus | Data Captured | Format | LLM Integration |
|------|-------|---------------|--------|-----------------|
| **llmdebug** | Test-time crash snapshots | Frames, locals, env, arrays | JSON/compact/TOON | Offline (snapshot -> LLM) |
| **LDB** (ACL 2024) | Verifying LLM-generated code | Execution traces | Structured logs | Iterative refinement |
| **traceback-with-variables** | Rich tracebacks | Frames + locals | Pretty text | None |
| **llm_exceptions** | Auto LLM invocation | Stack traces + code | Text -> LLM | Real-time |
| **stackprinter** | Log-based debugging | Frames + locals | Pretty text | None |
| **Rich** | Pretty tracebacks | Frames + locals | Terminal colors | None |

### IDE Integrations

| Tool | Error Context Features | Key Limitation |
|------|----------------------|----------------|
| **Cursor BugBot** | Runtime errors, call stack, terminal integration | No persistent snapshots |
| **GitHub Copilot** | Debugger state, deep analysis, iterative debugging | IDE-only, no pytest |
| **Windsurf** | RAG-based ~200K token context, console capture | Manual context curation |
| **JetBrains AI** | Agent traces, LangGraph workflow tracing | AI-app-specific |
| **Continue.dev** | Call stack analysis, open-source | No structured output |

### Commercial Products

| Product | Approach | Key Metric |
|---------|----------|------------|
| **Sentry Seer** | Production monitoring + autonomous AI fixing | 94.5% root cause accuracy |
| **Datadog Bits AI** | Distributed traces + AI agents (Dev, SRE, Security) | Enterprise-scale |
| **Rollbar** | Error grouping + stability scoring | Release tracking |

### llmdebug's Unique Strengths

1. **Zero-config pytest integration** - Just `pip install`, failures auto-capture
2. **LLM-optimized format** - Array summaries, compact JSON (~40%), TOON (~50%)
3. **Crash frame prioritization** - `frames[0]` is always the crash site
4. **ML/scientific computing focus** - NumPy, JAX, PyTorch, TensorFlow, CuPy detection
5. **Offline-first** - No external service, works in air-gapped environments
6. **A/B evaluation harness** - Empirical measurement of snapshot effectiveness

### Capability Snapshot

**Shipped:**
- Real-time IDE integration via MCP server
- Production crash capture (exception hooks, middleware)
- Snapshot diffing and comparison
- CLI for snapshot management
- Jupyter/notebook integration
- Automated hypothesis generation

**Open gaps:**
- Git bisect integration
- Cross-run failure clustering and searchable snapshot index

---

## 2. Academic Research on LLM Debugging

### Key Papers (2024-2026)

**LDB - Large Language Model Debugger (ACL 2024)**
- Tracks intermediate variable values after each basic block during execution
- **Result: 9.8% improvement** on HumanEval, MBPP, and TransCoder benchmarks
- **Key insight for llmdebug**: Capturing variable values at the crash site is directly
  validated as high-value information

**LLM4FL - Enhanced Fault Localization (2024)**
- Multi-agent system: Tester Agent + Debugger Agent with self-reflection
- Combines test code, stack trace data, and coverage data
- **Result: 19.27% improvement** over single-agent AutoFL
- **Key insight**: Multi-agent approaches with specialized roles outperform single-agent

**AutoCodeRover (ISSTA 2024)**
- AST-based autonomous program improvement with SBFL
- Patch validation loops leveraging test suites
- **Result: 22.33% on SWE-bench-lite, <$0.70/task, 701s average**
- **Key insight**: Iterative debugging with structured context beats single-shot

**SWE-Pruner (2025)**
- Line-level context pruning for code agent interactions
- Uses 0.6B parameter neural skimmer with <100ms latency
- **Result: 23-38% token reduction with equal/better performance**
- **Key insight**: Less focused context outperforms more unfocused context

### What Context Helps LLMs Most?

| Context Type | Impact | Evidence |
|-------------|--------|----------|
| Variable values at crash site | +9.1% accuracy | LDB (ACL 2024) |
| Stack traces + source code | Essential baseline | LLM4FL, AutoCodeRover |
| Structured format (JSON) | Better than text for debugging | Multiple studies |
| Focused context (5-10K tokens) | Beats large unfocused (50K+) | SWE-Pruner (2025) |
| CoT-structured analysis | +3.61% fix accuracy | Iterative APR (2025) |
| SBFL suspiciousness scores | +23% Top-1 improvement | LLM4FL (2024) |

### Gap Analysis from Research

1. **Problem complexity ceiling** - LLMs handle simple bugs well, struggle with multi-file architectural issues
2. **Over-reasoning** - Models generate excessive explanations that hinder clarity
3. **Context window limits** - SWE-bench codebases average 438K lines vs. limited context windows
4. **Hallucination in fixes** - LLMs invent non-existent variables, APIs, or logic
5. **Cost at scale** - Iterative debugging (which works best) is expensive
6. **Test suite dependency** - Most top systems require comprehensive tests

### Benchmarks

| Benchmark | Language | Size | Focus |
|-----------|----------|------|-------|
| SWE-bench | Python | 2,294 issues | Real GitHub issues (industry standard) |
| SWE-bench Verified | Python | 500 issues | Human-validated subset |
| SWE-bench Lite | Python | 300 issues | Tractable subset |
| Defects4J | Java | 835 bugs | Real bugs from 17 projects |
| BugsInPy | Python | 493 bugs | Python-specific faults |
| HumanEval | Python | 164 problems | Code generation + debugging |

---

## 3. Token Optimization Strategies

### Format Efficiency Comparison

| Format | Tokens (500-row dataset) | Accuracy/1K tokens | Best For |
|--------|-------------------------|--------------------| ---------|
| CSV | 48 (baseline) | N/A | Tabular data |
| TOON | 4,617 (61% less than JSON) | 26.9% | Uniform arrays |
| YAML | ~80% of JSON | 17.7% better than XML | Hierarchical configs |
| JSON Compact | 80 | N/A | Deep nesting |
| JSON Pretty | 111 (11,842 for dataset) | 15.3% | Human readability |

### Context Pruning Strategies from Production Tools

**SWE-Pruner (2025 - State of the Art):**
- 23-38% token reduction on multi-turn debugging tasks
- 14.84x compression on single-turn code tasks
- 64% success rate vs 62% baseline (fewer tokens = better performance)

**SWE-agent observation:**
- 67-76% of token budget spent on read/exploration operations
- Average SWE-bench codebase: 438K lines, issue description: ~195 words

**Aider repository map:**
- Default 1,000 tokens for repo context (configurable)
- Graph-ranking algorithm on file dependency graph

### Layered/Progressive Disclosure

Research-validated approach (Bug Knowledge Layer paper, 2025):

| Layer | Content | Token Budget | When |
|-------|---------|-------------|------|
| 1 | Exception + crash frame locals + array shapes | ~2K tokens | Always |
| 2 | Full stack trace + all locals + source context | ~5K tokens | On request |
| 3 | Environment + git context + related files | ~10K tokens | On request |

**Key finding**: Aggressive compression can backfire if the agent must re-fetch context.
Debugging sessions need **lower compression** than simple Q&A due to intricate state
dependencies.

### Variable Serialization Strategies

- **Large collections**: Show first 6 items + count (reprlib/cheap_repr pattern)
- **Nested objects**: Max depth of 3 levels, placeholder below
- **Non-serializable**: `<ClassName: state_summary>` (e.g., `<PostgresConnection: localhost:5432/db>`)
- **Size budget**: `sys.getsizeof()` check before serialization, reject >10MB objects
- **Structure preservation**: Never break JSON mid-string or leave unclosed brackets when truncating

---

## 4. MCP Server Integration

### Official Python SDK

**Package:** `mcp` (PyPI), by Anthropic
**Install:** `pip install "mcp[cli]"` or `uv add "mcp[cli]"`
**API:** FastMCP decorator-based, clean and Pythonic

### Proposed llmdebug MCP Server Design

```python
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("llmdebug")

@mcp.resource("llmdebug://snapshot/{snapshot_id}")
def get_snapshot(snapshot_id: str) -> str:
    """Read crash snapshot by ID (latest, timestamp, or hash)"""

@mcp.tool()
def llmdebug_list_snapshots(limit: int = 10) -> str:
    """List recent crash snapshots with timestamps and exception types"""

@mcp.tool()
def llmdebug_get_crash_frame(snapshot_id: str = "latest", frame_index: int = 0) -> str:
    """Get detailed frame info (locals, source code) for the crash site"""

@mcp.tool()
def llmdebug_analyze_array_shapes(snapshot_id: str = "latest") -> str:
    """Extract all array shapes from crash frame locals"""
```

### IDE Configuration Patterns

**Claude Code:**
```bash
claude mcp add --transport stdio llmdebug -- python -m llmdebug.mcp_server
```

**VS Code (`.vscode/mcp.json`):**
```json
{
  "servers": {
    "llmdebug": {
      "type": "stdio",
      "command": "uv",
      "args": ["run", "--with", "llmdebug[mcp]", "python", "-m", "llmdebug.mcp_server"]
    }
  }
}
```

### Best Practices

- **Service-prefix tools**: `llmdebug_*` prevents naming conflicts
- **Resources = read-only data** (snapshots), **Tools = actions** (analysis)
- **Return helpful error strings** instead of raising exceptions
- **Log to stderr only** (critical for stdio transport)
- **5-15 tools max** for discoverability
- **Flat arguments** with Literal types for enums

### Reference Implementations

- `markomanninen/mcp-debugpy` - Debugging-focused MCP server (closest analog)
- `samefarrar/mcp-pdb` - PDB debugger integration
- `modelcontextprotocol/python-sdk` - Official SDK with examples

---

## 5. Pytest Plugin Enhancements

### Additional Hooks to Implement

**`pytest_exception_interact`** - Fires when entering debugger on failure:
```python
@pytest.hookimpl(tryfirst=True)
def pytest_exception_interact(node, call, report):
    print("\n[llmdebug] Snapshot available at .llmdebug/latest.json")
```

**`pytest_fixture_setup`** - Track fixture setup timing and failures

### New Context to Capture

**Fixture values** (via `item.funcargs`):
```python
active_fixtures = {
    name: safe_repr(item.funcargs[name], cfg)
    for name in item.fixturenames
    if name in item.funcargs
}
pytest_context["fixtures"] = active_fixtures
```

**Markers** (via `item.iter_markers()`):
```python
pytest_context["markers"] = {
    mark.name: {"args": list(mark.args), "kwargs": dict(mark.kwargs)}
    for mark in item.iter_markers()
    if mark.name not in {"parametrize", "usefixtures"}
}
```

**Parameter indices** (via `callspec.indices`):
```python
if callspec is not None:
    pytest_context["param_indices"] = dict(callspec.indices)
```

**xdist worker detection:**
```python
if hasattr(item.config, "workerinput"):
    pytest_context["xdist_worker"] = item.config.workerinput["workerid"]
```

**Timeout configuration:**
```python
timeout_marker = item.get_closest_marker("timeout")
if timeout_marker:
    pytest_context["timeout_config"] = {
        "seconds": timeout_marker.args[0] if timeout_marker.args else None
    }
```

### Flaky Test Detection

Using pytest's Stash API for type-safe cross-hook storage:
```python
flaky_key = pytest.StashKey[list[str]]()

# In pytest_runtest_makereport:
outcomes = item.stash.setdefault(flaky_key, [])
outcomes.append(report.outcome)
if len(outcomes) > 1:
    pytest_context["rerun_count"] = len(outcomes) - 1
    pytest_context["flaky"] = outcomes[-1] == "passed" and "failed" in outcomes
```

### Innovative Plugins Worth Studying

- **inline-snapshot** (2025) - Generates expected values inline in tests
- **syrupy** - Modernized snapshot testing with multiple formats
- **pytest-split** - Intelligent test distribution by duration
- **pytest-testmon** - Smart test selection using Coverage.py

---

## 6. Differential Debugging & Snapshot Diffing

### Libraries for Snapshot Comparison

| Library | Best For | Key Features |
|---------|----------|-------------|
| **DeepDiff** | General object comparison | Tree views, delta objects, type-aware, numpy support |
| **dictdiffer** | Simple dict patching | Lightweight, patch/revert |
| **jsondiff** | JSON-specific diffs | Multiple diff syntaxes |

### DeepDiff Usage for Snapshots

```python
from deepdiff import DeepDiff

diff = DeepDiff(
    old_snapshot["frames"][0]["locals"],
    new_snapshot["frames"][0]["locals"],
    view='tree',
    ignore_order=False,
    significant_digits=6
)
```

### Git Bisect Integration

```bash
# Proposed CLI command:
llmdebug bisect --test=tests/test_foo.py::test_bar --good=v1.0 --bad=HEAD
```

Generates a `git bisect run` script that:
1. Runs the failing test at each commit
2. Checks `.llmdebug/latest.json` for exception type
3. Reports the first-bad-commit with snapshot context
4. Binary search: log2(20,000 commits) = ~15 checks

### Hypothesis Generation from Diffs

```python
def generate_hypotheses(snapshot: dict) -> list[dict]:
    crash_frame = snapshot["frames"][0]

    # Pattern: Empty arrays
    for var, data in crash_frame["locals"].items():
        if isinstance(data, dict) and data.get("shape", [None])[0] == 0:
            yield {"confidence": 0.9, "description": f"Empty array: {var}",
                   "suggestion": "Check upstream data loading/filtering"}

    # Pattern: NaN/Inf anomalies
    for var, data in crash_frame["locals"].items():
        if isinstance(data, dict) and data.get("anomalies"):
            yield {"confidence": 0.85, "description": f"Numeric anomalies in {var}",
                   "suggestion": "Check for division by zero or invalid ops"}
```

### Time-Travel Debugging Relationship

llmdebug uses **snapshotting at failure time** - a pragmatic hybrid:
- Zero runtime overhead (only captures on failure)
- Rich state capture (locals, arrays, git context)
- Limitation: No pre-failure state (addressed by `snapshot_section()` for checkpoints)

Enhancement: Incremental snapshots via multiple `snapshot_section()` calls provide
checkpoint-based time-travel within instrumented regions.

---

## 7. Production Crash Capture

### Exception Hook Pattern

```python
def install_hooks():
    """Install crash capture hooks for main thread and worker threads."""
    original_excepthook = sys.excepthook
    original_threading_hook = threading.excepthook

    def llmdebug_excepthook(exc_type, exc_value, exc_tb):
        if should_capture(exc_type, exc_value):
            capture_exception("main_thread", exc_value, exc_tb)
        original_excepthook(exc_type, exc_value, exc_tb)

    def llmdebug_threading_hook(args):
        if should_capture(type(args.exc_value), args.exc_value):
            capture_exception(f"thread_{args.thread.name}", args.exc_value, args.exc_traceback)
        original_threading_hook(args)

    sys.excepthook = llmdebug_excepthook
    threading.excepthook = llmdebug_threading_hook
```

**Key**: Always preserve and call the original hooks.

### Sampling Strategies

| Strategy | Description | Use Case |
|----------|-------------|----------|
| Always capture first occurrence | 100% for new error signatures | Development |
| Token bucket rate limiter | N captures/min per error type | Staging |
| Adaptive (Datadog-style) | Adjusts based on daily volumes | Production |

**Error signature**: Hash of `(exception_type, file, line)`.

### PII Scrubbing

**Default denylist** (following Sentry's pattern):
```python
PII_KEYS = {
    "password", "passwd", "secret", "api_key", "apikey",
    "access_token", "auth", "credentials", "csrf", "session",
    "cookie", "authorization", "token", "ssn", "credit_card"
}
```

**Optional advanced detection**: `scrubadub` for NER-based PII detection,
`presidio` for enterprise-grade (heavier dependencies).

### Async/Concurrent Capture

- **asyncio**: `loop.set_exception_handler()` for unhandled task exceptions
- **TaskGroup** (Python 3.11+): `except*` syntax for ExceptionGroup handling
- **ThreadPoolExecutor**: `future.add_done_callback()` pattern
- **ProcessPoolExecutor**: `multiprocessing.Queue` to send exceptions to parent

### WSGI/ASGI Middleware

```python
# Flask
app.wsgi_app = LLMDebugMiddleware(app.wsgi_app)

# FastAPI
app.add_middleware(LLMDebugASGIMiddleware)

# Django
MIDDLEWARE = ['llmdebug.middleware.DjangoMiddleware', ...]
```

### Memory Safety

- **Size budget**: `sys.getsizeof()` check before serialization, reject >10MB objects
- **reprlib depth limit**: `maxlevel=3` prevents deeply nested explosions
- **Compression option**: `.json.gz` output for disk savings (gzip, stdlib)
- Existing `max_items=50` and `max_str=500` truncation is already good

---

## 8. Jupyter & CLI Integration

### Jupyter/IPython Extension

**IPython magics:**
```python
@magics_class
class LLMDebugMagics(Magics):
    @line_magic
    def llmdebug_show(self, line):
        """Display latest crash snapshot"""

    @line_magic
    def llmdebug_list(self, line):
        """List available snapshots"""

    @cell_magic
    def llmdebug_section(self, line, cell):
        """Execute cell with snapshot instrumentation"""

def load_ipython_extension(ipython):
    ipython.register_magics(LLMDebugMagics)
```

**Usage:** `%load_ext llmdebug` then `%llmdebug_show`

**Custom exception handler:**
```python
ip = get_ipython()
ip.set_custom_exc((Exception,), llmdebug_exception_handler)
```

**Widget display:** Accordion with Output widgets for collapsible stack frames,
`_repr_html_()` protocol for rich inline rendering.

### CLI Framework (Typer + Rich)

**Recommended:** Typer for type-hint-driven CLI, Rich for terminal formatting.

```bash
llmdebug show [latest]        # Render snapshot with syntax highlighting
llmdebug list [--limit 10]    # Table of recent failures
llmdebug diff snap1 snap2     # Structural diff of two snapshots
llmdebug clean [--keep 5]     # Remove old snapshots
llmdebug bisect --test=...    # Git bisect integration
llmdebug mcp install vscode   # Generate MCP config
```

**Rich output features:**
- `Tree` for stack frame visualization
- `Table` for local variables with type columns
- `Syntax` for highlighted source code context
- `Panel` for exception info header
- `Layout` for multi-panel snapshot views

### VS Code Notebook Integration

- Debug Adapter Protocol (DAP) for cell debugging
- Custom MIME type renderers for error display
- Reference: `vscode-simple-jupyter-notebook` by Microsoft

---

## 9. Eval Framework Improvements

### Current State

- ~30 hand-crafted cases across 8 categories
- A/B testing: `traceback_only` vs `with_snapshot`
- Tracks: success rate, attempts, token usage, runtime
- Multiple patcher backends (OpenAI-compatible APIs)

### Expansion Strategy

**Target: 100-200 cases for statistical power**

| Source | Cases | Effort |
|--------|-------|--------|
| Hand-crafted (current) | 30 | Already done |
| Cosmic Ray mutation testing | 50-100 | Medium (semi-automated) |
| BugsInPy real-world bugs | 20-30 | Medium (curation needed) |
| Community contributions | 10-20 | Low (with validation CI) |

### Mutation Testing for Case Generation

**Cosmic Ray** (most mature Python mutation tester):
```bash
cosmic-ray init cosmic-ray.toml src/llmdebug
cosmic-ray exec cosmic-ray.toml
# Extract live mutants as eval cases
```

**PyTation** (Python-specific realistic faults):
- 7 mutation operators targeting idiomatic Python bugs
- Mutable default arguments, missing `await`, incorrect exception handling

### Statistical Rigor

**Add to `analyze_results.py`:**
```python
from scipy.stats import ttest_rel
import numpy as np

def compute_uplift_significance(paired_data):
    before = [x[0] for x in paired_data]  # traceback_only results
    after = [x[1] for x in paired_data]   # with_snapshot results
    t_stat, p_value = ttest_rel(before, after)
    diffs = [a - b for a, b in zip(after, before)]
    cohens_d = np.mean(diffs) / np.std(diffs) if np.std(diffs) > 0 else 0
    return {"p_value": p_value, "cohens_d": cohens_d,
            "effect_size": "small" if abs(cohens_d) < 0.5
                      else "medium" if abs(cohens_d) < 0.8
                      else "large"}
```

**Sample size guidance:**
- 30 cases: Exploratory, directional signals only
- 100 cases per condition: 80% power for medium effects (d=0.5)
- 300 cases total: 90% power, publication-grade

### Additional Metrics

- **Patch quality score** (1-5) via LLM-as-judge when tests pass
- **Cohen's d** effect size for uplift significance
- **Confidence intervals** around success rates
- Multiple sampling (k=3-5 attempts per case) to reduce stochastic variance

### Enhanced Case Metadata

```json
{
  "category": "hidden_state",
  "difficulty": "medium",
  "difficulty_rationale": "Crash frame shows None.call() but root cause is upstream",
  "snapshot_dependency": "required",
  "expected_fix_lines": 3,
  "distractor_lines": 12,
  "bug_source": "synthetic",
  "requires_domain_knowledge": ["asyncio"],
  "discoverability": "requires_inspection"
}
```

---

## 10. Developer Ecosystem & Adoption

### Market Context (2025-2026)

- **92% of developers** use AI tools regularly (JetBrains Developer Ecosystem 2025)
- **45%** use AI specifically for debugging assistance
- Software dev tools market: **$6.41B (2025) -> $13.70B (2030)** at 16.4% CAGR
- Top 3 AI coding tools by revenue: GitHub Copilot, Claude Code, Cursor (each $1B+ ARR)

### Successful pytest Plugin Patterns

| Plugin | Downloads/month | Success Factor |
|--------|----------------|----------------|
| pytest-cov | 87.7M | One-liner usage, removes boilerplate |
| pytest-xdist | 60.3M | Single flag (`-n auto`), auto-scales |
| pytest-asyncio | 58.9M | Fills critical gap, async support |
| pytest-mock | 50.7M | Simplified interface over unittest.mock |

**Common patterns:** Zero-config defaults, focused solutions (one pain point), composability
with other plugins, comprehensive README.

### Documentation Strategy

**Follow FastAPI's pattern:**
```
Homepage (value proposition)
  -> Quick Start (3-step: install -> run -> read)
    -> Integration Guides (pytest, decorator, context manager)
      -> API Reference (config, hooks, output formats)
        -> Community (contributing, discussions)
```

**Key metrics to optimize:**
- Time to first success: <5 minutes
- Self-validating: Snapshot output proves the tool works
- Before/after comparison showing hypothesis-driven debugging

### PyPI Packaging Strategy

```toml
[project.optional-dependencies]
cli = ["click>=8.0", "rich>=13.0"]
mcp = ["mcp>=1.0"]
toon = ["toons>=0.1"]
jupyter = ["ipython>=8.0"]
evals = ["datasets>=2.0"]
dev = ["pytest>=7.0", "..."]
```

Core package stays dependency-light (`filelock` only).

### AI Coding Tool Integration

- **CLAUDE.md template** for projects using llmdebug (already implemented)
- **MCP server** for Claude Code, Cursor, VS Code Copilot integration
- **README positioning**: "Built for AI-Assisted Debugging" section
- **Integration guides** per tool: Claude Code workflow, Cursor workflow

### Community Strategy

| Phase | Actions |
|-------|---------|
| Foundation | GitHub Discussions, GitHub Sponsors, CONTRIBUTING.md |
| Growth | Discord server (>100 users), blog posts, integration guides |
| Conference | PyCon 2026 talk: "Debugging in the AI Era" |
| Sustainability | Corporate sponsors, GitHub Accelerator, Open Source Pledge |

### Monetization (Long-Term)

**Open-core model:**
- **Free**: pytest plugin, decorator, context manager, CLI, MCP server
- **Paid**: Team snapshot collaboration, historical search, CI/CD dashboard,
  LLM-powered pattern detection, enterprise SSO/compliance

---

## Prioritized Improvement Roadmap

### Tier 1: High-Impact, Low-Effort (Next Release)

| # | Feature | Effort | Impact | Evidence |
|---|---------|--------|--------|----------|
| 1 | **Triage-first CLI/MCP workflow** (regression-oriented summaries) | 2-4 days | Highest | Faster root-cause loops for agents + humans |
| 2 | **Cross-run failure clustering** (eval + snapshot signatures) | 3-5 days | High | Better analysis and RCA prioritization |
| 3 | **Shareable report artifact** (single-file HTML/JSON) | 2-3 days | High | Easier adoption and team collaboration |

### Tier 2: High-Impact, Medium-Effort (v2.4+)

| # | Feature | Effort | Impact |
|---|---------|--------|--------|
| 4 | **Layered snapshot disclosure** (3 layers) | 3-5 days | High |
| 5 | **Root-cause ranking** (stack + locals + coverage signals) | 5-7 days | High |

### Tier 3: Medium-Impact, Higher-Effort (v3.0+)

| # | Feature | Effort | Impact |
|---|---------|--------|--------|
| 6 | **Eval framework expansion** (100+ cases, richer contamination controls) | 2-4 weeks | Strategic |
| 7 | **Git bisect integration** | 2-3 days | Medium |
| 8 | **Enhanced git context** (blame, recent commits, diffs) | 2-3 days | Medium |
| 9 | **Historical snapshot index/search** | 3-5 days | Medium |

### Tier 4: Long-Term Vision

| # | Feature | Effort | Impact |
|---|---------|--------|--------|
| 13 | Cross-language SDKs (JS, Rust, Go) | Months | Strategic |
| 14 | Automated repair mode (`llmdebug fix`) | Weeks | High |
| 15 | Distributed tracing (Ray, Dask, multiprocessing) | Weeks | Medium |

---

## Packaging Strategy

```toml
[project.optional-dependencies]
cli = ["click>=8.0", "rich>=13.0"]         # CLI commands
mcp = ["mcp>=1.0"]                         # MCP server for IDEs
toon = ["toons>=0.1"]                      # TOON output format
jupyter = ["ipython>=8.0"]                 # Notebook integration
evals = ["datasets>=2.0"]                  # Eval case import tooling
dev = ["pytest>=7.0", "..."]               # Local dev/test tooling

[project.scripts]
llmdebug = "llmdebug.cli:main"             # CLI entry point
llmdebug-mcp = "llmdebug.mcp_server:main"  # MCP server entry point
```

**Install patterns:**
```bash
pip install llmdebug               # pytest plugin only (zero extra deps)
pip install llmdebug[cli]          # + CLI for manual snapshot management
pip install llmdebug[mcp]          # + MCP server for Claude Code / Cursor
pip install llmdebug[jupyter]      # + Jupyter/IPython integration
pip install llmdebug[evals]        # + eval importer dependencies
```

---

## Research-Backed Claims

These claims are directly supported by 2024-2026 academic papers and can be cited:

1. **"Runtime variable values improve LLM debugging by 9%+"**
   - Source: LDB (ACL 2024), HumanEval benchmark

2. **"Focused context outperforms large unfocused dumps"**
   - Source: SWE-Pruner (2025), 23-38% fewer tokens with equal/better accuracy

3. **"Structured crash snapshots match state-of-the-art debugging methodology"**
   - Source: AutoCodeRover (ISSTA 2024), iterative context refinement

4. **"CoT-structured analysis improves fix accuracy by 3.61%"**
   - Source: Iterative APR with execution traces (2025)

5. **"Compact format saves 30-60% tokens vs. formatted JSON"**
   - Source: TOON benchmark studies (multiple, 2025)

6. **"Multi-agent debugging with specialized roles improves accuracy by 19.27%"**
   - Source: LLM4FL (2024), Tester + Debugger agent architecture

---

## Sources

### Academic Papers
- [LDB - Debug like a Human (ACL 2024)](https://arxiv.org/html/2402.16906v6)
- [LLM4FL - Enhancing FL with Ordered Analysis (2024)](https://arxiv.org/html/2409.13642v1)
- [AutoCodeRover - Autonomous Program Improvement (ISSTA 2024)](https://arxiv.org/html/2404.05427v2)
- [SWE-Pruner - Self-Adaptive Context Pruning (2025)](https://arxiv.org/html/2601.16746v1)
- [SWE-bench - Can LMs Resolve GitHub Issues? (ICLR 2024)](https://arxiv.org/pdf/2310.06770)
- [AutoFL - LLM-Based Explainable Fault Localization (FSE 2024)](https://2024.esec-fse.org/details/fse-2024-research-papers/15/)
- [LLMAO - Test-Free Fault Localization (ICSE 2024)](https://aidanby.github.io/files/icse24.pdf)
- [Bug Fixing with Layered Knowledge Injection (2025)](https://arxiv.org/html/2506.24015v1)
- [Novice Program Fault Localization (Dec 2025)](https://arxiv.org/abs/2512.03421)
- [DoVer - Intervention-Driven Multi-Agent Debugging (2024)](https://arxiv.org/html/2512.06749v1)
- [Hybrid Fault-Driven Mutation Testing for Python (PyTation)](https://arxiv.org/html/2601.19088)

### Tools & Libraries
- [MCP Python SDK](https://github.com/modelcontextprotocol/python-sdk)
- [mcp-debugpy - Debugging MCP Server](https://github.com/markomanninen/mcp-debugpy)
- [DeepDiff](https://github.com/seperman/deepdiff)
- [Cosmic Ray - Python Mutation Testing](https://cosmic-ray.readthedocs.io/)
- [Rich - Terminal Formatting](https://rich.readthedocs.io/)
- [Typer - CLI Framework](https://typer.tiangolo.com/)
- [Sentry Python SDK](https://github.com/getsentry/sentry-python)
- [stackprinter](https://github.com/cknd/stackprinter)
- [traceback-with-variables](https://github.com/andy-landy/traceback_with_variables)
- [LLMDebugger (LDB)](https://github.com/FloridSleeves/LLMDebugger)

### Industry & Market
- [JetBrains Developer Ecosystem 2025](https://devecosystem-2025.jetbrains.com/)
- [Model Context Protocol Announcement](https://www.anthropic.com/news/model-context-protocol)
- [VS Code MCP Server Configuration](https://code.visualstudio.com/docs/copilot/customization/mcp-servers)
- [TOON vs JSON Token Efficiency](https://www.tensorlake.ai/blog/toon-vs-json)
- [AI Code Assistant Market Report](https://www.mordorintelligence.com/industry-reports/artificial-intelligence-code-tools-market)

### Pytest Ecosystem
- [pytest API Reference](https://docs.pytest.org/en/stable/reference/reference.html)
- [pytest Hooks Documentation](https://docs.pytest.org/en/stable/_modules/_pytest/hookspec.html)
- [pytest-timeout](https://github.com/pytest-dev/pytest-timeout)
- [pytest-xdist](https://github.com/pytest-dev/pytest-xdist)
- [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures)
- [inline-snapshot](https://github.com/15r10nk/inline-snapshot)

### Documentation & Community
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [Building Open Source Community (GitHub)](https://github.blog/open-source/maintainers/four-steps-toward-building-an-open-source-community/)
- [Open Source Funding 2025](https://www.eliostruyf.com/whos-funding-open-source-2025-guide-maintainers/)
- [Contextual Retrieval (Anthropic)](https://www.anthropic.com/news/contextual-retrieval)
