# Root Cause Analysis: node_modules Still Being Indexed Despite force_include_paths Fix

**Date**: 2026-02-02
**Project**: mcp-vector-search
**Issue**: node_modules directories are still being indexed when using `force_include_paths: ["repos/"]` despite recent fix in commit 3e0f6d0

---

## Executive Summary

**Root Cause**: The fix in commit 3e0f6d0 was applied to `ProjectManager._should_ignore_path()` in `project.py`, but there is a **duplicate implementation** of the same logic in `FileDiscovery.should_ignore_path()` in `file_discovery.py` that was NOT updated. The indexer uses `FileDiscovery` class, not `ProjectManager`, so the fix never took effect during indexing.

**Impact**: Users with `force_include_paths` configured are still indexing `node_modules`, `.git`, `__pycache__`, and other DEFAULT_IGNORE_PATTERNS directories, leading to:
- Extremely slow indexing (250K+ files in node_modules)
- Memory exhaustion
- Polluted search results
- Storage waste

**Fix Required**: Apply the same DEFAULT_IGNORE_PATTERNS priority fix to `FileDiscovery.should_ignore_path()` in `file_discovery.py`.

---

## Evidence

### 1. User Report
User reported seeing files like:
```
repos/duetto-mcp/node_modules/.pnpm/@babel+...
repos/llm-integration-factory/node_modules/.package-lock.json
```

in their index despite having version 2.1.8 installed (which includes the fix from commit 3e0f6d0).

### 2. Configuration
**File**: `/Users/masa/Clients/Duetto/CTO/.mcp-vector-search/config.json`
```json
{
  "force_include_paths": ["repos/"],
  "respect_gitignore": true
}
```

### 3. Index Metadata Evidence
**File**: `/Users/masa/Clients/Duetto/CTO/.mcp-vector-search/index_metadata.json`

Contains thousands of entries like:
```json
"/Users/masa/Clients/Duetto/CTO/repos/llm-integration-factory/node_modules/.package-lock.json": 1768774379.9489155,
"/Users/masa/Clients/Duetto/CTO/repos/llm-integration-factory/node_modules/@cspotcode/source-map-support/LICENSE.md": 1768774379.8749275,
```

### 4. Indexing Timeline
- **Commit 3e0f6d0** (fix): 2026-02-02 21:37:59 EST
- **Package installed**: 2026-02-02 22:28:04 EST (after fix)
- **Indexing run**: 2026-02-02 22:29:29 EST (using v2.1.8)
- **Index updated**: 2026-02-02 22:32:44 EST

The indexing run occurred AFTER the fix was installed, yet node_modules was still indexed.

---

## Code Analysis

### The Fix That Was Applied (commit 3e0f6d0)

**File**: `src/mcp_vector_search/core/project.py`
**Method**: `ProjectManager._should_ignore_path()`

```python
def _should_ignore_path(self, path: Path, is_directory: bool | None = None) -> bool:
    # FIRST: Check DEFAULT_IGNORE_PATTERNS (node_modules, __pycache__, etc.)
    # These are ALWAYS ignored, even inside force_include_paths
    # This prevents accidentally indexing node_modules with 250K+ files
    for part in path.parts:
        if part in DEFAULT_IGNORE_PATTERNS:
            return True  # ← Returns early, BEFORE force_include_paths check

    # Check relative path from project root
    try:
        relative_path = path.relative_to(self.project_root)
        for part in relative_path.parts:
            if part in DEFAULT_IGNORE_PATTERNS:
                return True  # ← Returns early
    except ValueError:
        return True

    # SECOND: Check force_include_paths - they override gitignore (but not default ignores)
    if config and config.force_include_paths:
        # ... force_include_paths logic ...
```

**Fix Summary**: DEFAULT_IGNORE_PATTERNS check moved BEFORE force_include_paths check.

---

### The Unfixed Code (STILL BROKEN)

**File**: `src/mcp_vector_search/core/file_discovery.py`
**Method**: `FileDiscovery.should_ignore_path()`
**Lines**: 327-449

```python
def should_ignore_path(self, file_path: Path, is_directory: bool | None = None) -> bool:
    """Check if a path should be ignored."""

    # ... cache check ...

    try:
        relative_path = file_path.relative_to(self.project_root)
        relative_path_str = str(relative_path).replace("\\", "/")

        # 0. Check force_include_paths FIRST - they override everything  ← BUG!
        if self.config and self.config.force_include_paths:
            for include_path in self.config.force_include_paths:
                include_path_normalized = include_path.rstrip("/")

                # Check if path starts with include_path
                if relative_path_str.startswith(include_path_normalized + "/"):
                    logger.debug(
                        f"Force-including {relative_path} (inside force_include_path: {include_path})"
                    )
                    self._ignore_path_cache[cache_key] = False
                    return False  # Don't ignore  ← RETURNS HERE FOR repos/*/node_modules!

        # ... later in the function ...

        # 3. Check each part of the path against default ignore patterns  ← NEVER REACHED!
        for part in relative_path.parts:
            for pattern in self._ignore_patterns:
                if fnmatch.fnmatch(part, pattern):
                    logger.debug(
                        f"Path ignored by pattern '{pattern}' matching '{part}': {file_path}"
                    )
                    self._ignore_path_cache[cache_key] = True
                    return True
```

**Problem**: The comment at line 351 says "they override everything", and force_include_paths is checked FIRST, before DEFAULT_IGNORE_PATTERNS. This is the OLD behavior that was supposed to be fixed!

---

### Which Code Path is Actually Used?

**File**: `src/mcp_vector_search/core/indexer.py`

```python
class SemanticIndexer:
    def __init__(self, project_root: Path, config: ProjectConfig):
        # Initialize helper classes
        self.file_discovery = FileDiscovery(  # ← Uses FileDiscovery class!
            project_root=project_root,
            file_extensions=file_extensions_set,
            config=config,
        )

    async def index_project(self, ...):
        # Find all indexable files
        all_files = self.file_discovery.find_indexable_files()  # ← Calls FileDiscovery
```

**Conclusion**: The indexer uses `FileDiscovery`, NOT `ProjectManager`. Therefore, the fix in `ProjectManager._should_ignore_path()` has NO EFFECT on indexing behavior.

---

## Execution Trace Example

**Path**: `repos/llm-integration-factory/node_modules/.package-lock.json`
**Config**: `force_include_paths: ["repos/"]`

### Expected Behavior (with fix):
1. ✓ Check DEFAULT_IGNORE_PATTERNS → `node_modules` found → return True (ignore)
2. ✗ force_include_paths check → never reached
3. **Result**: File NOT indexed

### Actual Behavior (current code):
1. ✓ Check force_include_paths → `repos/llm-integration-factory/node_modules/.package-lock.json` starts with `repos/` → return False (don't ignore)
2. ✗ DEFAULT_IGNORE_PATTERNS check → never reached
3. **Result**: File IS indexed ❌

---

## Why This Happened

### Code Duplication
There are TWO separate implementations of `should_ignore_path()`:

1. **`ProjectManager._should_ignore_path()`** in `project.py`
   - Used by: `ProjectManager._iter_source_files()`
   - Purpose: Language detection, file counting for project info
   - Status: **FIXED** ✓

2. **`FileDiscovery.should_ignore_path()`** in `file_discovery.py`
   - Used by: `SemanticIndexer` via `FileDiscovery` class
   - Purpose: Actual indexing file discovery
   - Status: **NOT FIXED** ❌

### Why The Duplication Exists

Looking at the git history:
- `ProjectManager` was the original implementation
- `FileDiscovery` was likely extracted as a separate class for better separation of concerns
- The logic was duplicated instead of refactored to share the same code
- When the fix was applied to `ProjectManager`, `FileDiscovery` was overlooked

---

## Recommended Fix

### Option 1: Apply Same Fix to FileDiscovery (Quick Fix)

**File**: `src/mcp_vector_search/core/file_discovery.py`
**Method**: `should_ignore_path()`

Move DEFAULT_IGNORE_PATTERNS check to BEFORE force_include_paths check:

```python
def should_ignore_path(self, file_path: Path, is_directory: bool | None = None) -> bool:
    """Check if a path should be ignored."""

    # Check cache first
    cache_key = str(file_path)
    if cache_key in self._ignore_path_cache:
        return self._ignore_path_cache[cache_key]

    try:
        relative_path = file_path.relative_to(self.project_root)
        relative_path_str = str(relative_path).replace("\\", "/")

        # 0. FIRST: Check DEFAULT_IGNORE_PATTERNS (ALWAYS ignored)
        # These are NEVER force-included, even with force_include_paths
        for part in relative_path.parts:
            for pattern in self._ignore_patterns:
                if fnmatch.fnmatch(part, pattern):
                    logger.debug(
                        f"Path ignored by DEFAULT pattern '{pattern}' matching '{part}': {file_path}"
                    )
                    self._ignore_path_cache[cache_key] = True
                    return True

        # 1. Check force_include_paths - they override gitignore (but not default ignores)
        if self.config and self.config.force_include_paths:
            # ... existing force_include_paths logic ...

        # ... rest of the function ...
```

### Option 2: Refactor to Share Logic (Better Long-Term)

Extract `should_ignore_path` logic into a shared utility function to prevent future divergence:

**New File**: `src/mcp_vector_search/utils/path_filtering.py`

```python
def should_ignore_path(
    path: Path,
    project_root: Path,
    config: ProjectConfig | None,
    ignore_patterns: set[str],
    gitignore_parser: GitignoreParser | None,
    is_directory: bool | None = None,
) -> bool:
    """Shared path filtering logic for both ProjectManager and FileDiscovery."""
    # ... unified implementation ...
```

Then have both `ProjectManager` and `FileDiscovery` call this shared function.

---

## Testing Recommendations

### Test Case 1: node_modules with force_include_paths
```yaml
force_include_paths:
  - "repos/"
```

**Expected**: `repos/*/node_modules/**` should NOT be indexed

### Test Case 2: .git with force_include_paths
```yaml
force_include_paths:
  - "repos/"
```

**Expected**: `repos/*/.git/**` should NOT be indexed

### Test Case 3: __pycache__ with force_include_paths
```yaml
force_include_paths:
  - "src/"
```

**Expected**: `src/**/__pycache__/**` should NOT be indexed

### Test Case 4: Verify force_include_paths still works for legitimate files
```yaml
force_include_paths:
  - "repos/"
```

**Expected**: `repos/**/*.py` and `repos/**/*.js` SHOULD be indexed (non-ignored files)

---

## Priority

**CRITICAL** - This bug causes severe performance degradation and makes the tool nearly unusable for projects with:
- Multiple node_modules directories
- Monorepos with subprojects
- Any project using force_include_paths

**Impact Scope**:
- Affects all users using `force_include_paths` configuration
- Silently indexes hundreds of thousands of unwanted files
- Wastes disk space, memory, and indexing time
- Pollutes search results with irrelevant code

---

## Related Files

**To Fix**:
- `src/mcp_vector_search/core/file_discovery.py` (primary fix location)

**Verification Files**:
- `src/mcp_vector_search/core/project.py` (reference implementation with fix)
- `src/mcp_vector_search/config/defaults.py` (DEFAULT_IGNORE_PATTERNS definition)

**Test Files**:
- Add test coverage for `FileDiscovery.should_ignore_path()` with force_include_paths
- Ensure DEFAULT_IGNORE_PATTERNS override force_include_paths in all scenarios

---

## Conclusion

The fix in commit 3e0f6d0 was incomplete. While it correctly updated `ProjectManager._should_ignore_path()`, it missed the duplicate implementation in `FileDiscovery.should_ignore_path()` which is the actual code path used during indexing. This is a classic case of code duplication leading to partial fixes and inconsistent behavior.

**Next Steps**:
1. Apply the same fix to `FileDiscovery.should_ignore_path()`
2. Add test coverage to prevent regression
3. Consider refactoring to eliminate code duplication (longer-term improvement)
4. Update commit 3e0f6d0's commit message or release notes to indicate incomplete fix
