# Chunk ID Content Hashing Fix Locations

**Date**: 2025-01-29
**Investigation**: chunk_id generation using _get_line_range vs node.text.decode()
**Issue**: Minified files cause different content extraction between _get_line_range (splits lines) and tree-sitter nodes (preserves original text), leading to chunk_id hash mismatches.

---

## Executive Summary

The chunk_id is generated by hashing the first 100 characters of `content` in `CodeChunk.__post_init__()`. The problem occurs because:

1. **Content is extracted using `_get_line_range()`** in all parsers
2. **`_get_line_range()` splits content into lines** using `content.splitlines(keepends=True)`
3. **For minified files**, this line-splitting creates different content than tree-sitter's `node.text.decode()`
4. **The hash computed on split-line content** doesn't match the hash that would be computed on the original AST node text
5. **Result**: Duplicate chunks for the same function/class in minified files

---

## Root Cause Analysis

### Current Flow

```
Parser extracts node
  ↓
Calls _get_line_range(lines, start_line, end_line)
  ↓
utils.get_line_range() joins lines[start:end]
  ↓
Content passed to CodeChunk()
  ↓
CodeChunk.__post_init__() generates chunk_id
  ↓
Hashes content[:100] for chunk_id
```

### The Problem

**For normal files:**
- Tree-sitter node boundaries align with lines
- `_get_line_range()` and `node.text.decode()` produce identical content
- chunk_id hash is consistent

**For minified files:**
- Tree-sitter node boundaries don't align with lines (entire file is 1-2 lines)
- `_get_line_range(lines, 1, 1)` returns entire minified file
- `node.text.decode()` returns only the specific function/class AST node
- **Different content = different hash = different chunk_id**

---

## Key File Locations

### 1. Chunk ID Generation (Where Hash is Computed)

**File**: `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/core/models.py`

**Lines 57-66**:
```python
# Generate chunk ID if not provided
if self.chunk_id is None:
    import hashlib

    # Include name and first 50 chars of content for uniqueness
    # This ensures deterministic IDs while handling same-location chunks
    name = self.function_name or self.class_name or ""
    content_hash = hashlib.sha256(self.content[:100].encode()).hexdigest()[:8]
    id_string = f"{self.file_path}:{self.chunk_type}:{name}:{self.start_line}:{self.end_line}:{content_hash}"
    self.chunk_id = hashlib.sha256(id_string.encode()).hexdigest()[:16]
```

**Key Issue**: The hash uses `self.content[:100]` which comes from the parser's extracted content.

---

### 2. Content Extraction Utility (Line-Based)

**File**: `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/parsers/utils.py`

**Lines 27-42**:
```python
def get_line_range(lines: list[str], start_line: int, end_line: int) -> str:
    """Get content from a range of lines.

    Args:
        lines: List of lines
        start_line: Starting line number (1-indexed)
        end_line: Ending line number (1-indexed, inclusive)

    Returns:
        Joined content from the line range
    """
    # Convert to 0-indexed
    start_idx = max(0, start_line - 1)
    end_idx = min(len(lines), end_line)

    return "".join(lines[start_idx:end_idx])
```

**Key Issue**: This joins lines by index, which for minified files returns entire file instead of specific node.

---

### 3. Parser Content Extraction (All Languages)

All parsers follow the same pattern of calling `_get_line_range()`:

#### **Python Parser** (via helpers)

**File**: `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/parsers/python_helpers/node_extractors.py`

**Lines 42-47** (FunctionExtractor):
```python
function_name = MetadataExtractor.get_node_name(node)
start_line = node.start_point[0] + 1
end_line = node.end_point[0] + 1

# Get function content
content = self.base_parser._get_line_range(lines, start_line, end_line)
```

**Similar pattern in**:
- `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/parsers/python_helpers/fallback_parser.py:57`
- `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/parsers/python_helpers/fallback_parser.py:86`

#### **JavaScript/TypeScript Parser**

**File**: `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/parsers/javascript.py`

**Lines 142-145** (_parse_function_declaration):
```python
start_line = node.start_point[0] + 1
end_line = node.end_point[0] + 1

content = self._get_line_range(lines, start_line, end_line)
```

**All occurrences in javascript.py**:
- Line 145: `content = self._get_line_range(lines, start_line, end_line)`
- Line 186: `content = self._get_line_range(lines, start_line, end_line)`
- Line 226: `content = self._get_line_range(lines, start_line, end_line)`
- Line 265: `content = self._get_line_range(lines, start_line, end_line)`
- Line 295: `content = self._get_line_range(lines, start_line, end_line)`
- Line 460: `func_content = self._get_line_range(lines, start_line, end_line)`
- Line 487: `class_content = self._get_line_range(lines, start_line, end_line)`
- Line 515: `interface_content = self._get_line_range(lines, start_line, end_line)`

#### **Dart Parser**

**File**: `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/parsers/dart.py`

**10 occurrences**:
- Line 154, 184, 213, 243, 272, 356, 399, 429, 468

#### **PHP Parser**

**File**: `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/parsers/php.py`

**13 occurrences**:
- Line 176, 216, 246, 280, 314, 348, 458, 493, 526, 573

#### **Ruby Parser**

**File**: `/Users/masa/Projects/mcp-vector-search/src/mcp_vector_search/parsers/ruby.py`

**2 occurrences**:
- Line 171, 209

---

## The Fix Required

### Current Implementation

```python
# In all parsers (e.g., javascript.py:145)
start_line = node.start_point[0] + 1
end_line = node.end_point[0] + 1
content = self._get_line_range(lines, start_line, end_line)
```

### Required Change

```python
# Use tree-sitter's native text extraction
content = node.text.decode()
# No need for start_line/end_line for content extraction
# (still needed for CodeChunk metadata)
```

### Why This Fixes It

- **`node.text.decode()`** extracts exact bytes from the AST node
- **Works identically for minified and non-minified files**
- **Always returns the specific function/class code**, not entire file
- **Deterministic**: Same node = same content = same hash = same chunk_id

---

## Files That Need Changes

### High Priority (Tree-sitter Parsers)

All tree-sitter based parsers need to change from `_get_line_range()` to `node.text.decode()`:

1. **Python Parser Helper** (`python_helpers/node_extractors.py`):
   - Line 47: FunctionExtractor
   - Line 154: Import extraction (if using nodes)

2. **Python Fallback** (`python_helpers/fallback_parser.py`):
   - Line 57: Function extraction
   - Line 86: Class extraction

3. **JavaScript Parser** (`parsers/javascript.py`):
   - Lines: 145, 186, 226, 265, 295, 460, 487, 515

4. **Dart Parser** (`parsers/dart.py`):
   - Lines: 154, 184, 213, 243, 272, 356, 399, 429, 468

5. **PHP Parser** (`parsers/php.py`):
   - Lines: 176, 216, 246, 280, 314, 348, 458, 493, 526, 573

6. **Ruby Parser** (`parsers/ruby.py`):
   - Lines: 171, 209

### Low Priority (Fallback/Text Parsers)

These don't use tree-sitter nodes, so they keep using `_get_line_range()`:

- **Base Parser** (`parsers/base.py:280`): Fallback chunking
- **Text Parser** (`parsers/text.py`): Line-based chunking
- **Utils** (`parsers/utils.py`): Helper functions for non-tree-sitter parsers

---

## Implementation Strategy

### Option 1: Minimal Change (Recommended)

Change each parser's content extraction to use `node.text.decode()`:

**Example for JavaScript parser** (`javascript.py:145`):

**Before**:
```python
def _parse_function_declaration(self, node, lines, file_path, class_name=None):
    start_line = node.start_point[0] + 1
    end_line = node.end_point[0] + 1

    content = self._get_line_range(lines, start_line, end_line)
    # ... rest of function
```

**After**:
```python
def _parse_function_declaration(self, node, lines, file_path, class_name=None):
    start_line = node.start_point[0] + 1
    end_line = node.end_point[0] + 1

    # Use tree-sitter's native text extraction for consistent hashing
    content = node.text.decode()
    # ... rest of function
```

**Pros**:
- Minimal code change
- Fixes chunk_id duplication for minified files
- No change to non-tree-sitter parsers

**Cons**:
- Need to change ~40 lines across 6 files
- Still passes `lines` parameter (unused for content, but used for docstring extraction)

### Option 2: Refactor Extractors

Create a helper method in BaseParser:

```python
# In base.py
def _extract_node_content(self, node) -> str:
    """Extract content from tree-sitter node.

    Uses node.text.decode() for deterministic content extraction
    that works consistently with minified files.
    """
    return node.text.decode()
```

Then update all parsers to call:
```python
content = self._extract_node_content(node)
```

**Pros**:
- Centralized logic
- Clear intent
- Easy to modify behavior later

**Cons**:
- More extensive changes
- Need to ensure `node` is available in all contexts

---

## Testing Strategy

### Test Cases Required

1. **Minified JavaScript file with multiple functions**:
   - Verify each function gets unique chunk_id
   - Verify no duplicate chunks in database
   - Verify chunk_id is deterministic across re-indexing

2. **Normal JavaScript file**:
   - Verify chunk_id generation still works
   - Verify no regression in content extraction

3. **Python, PHP, Ruby, Dart files**:
   - Same tests as JavaScript

4. **Edge cases**:
   - Empty functions
   - Functions on same line (minified)
   - Very long functions (>100 chars)
   - Unicode characters in function bodies

### Validation Commands

```bash
# Index minified file
mcp-vector-search index path/to/minified.js

# Check for duplicate chunk_ids
sqlite3 .mcp-vector-search/vector_store.db \
  "SELECT chunk_id, COUNT(*) FROM chunks GROUP BY chunk_id HAVING COUNT(*) > 1"

# Should return no results after fix
```

---

## Related Issues

**Previous Fix**: Commit `4af20e1` changed from using `id` (location-based) to `chunk_id` (content-hash-based) to prevent duplicates.

**This Fix**: Ensures the content used for chunk_id hashing is extracted consistently using `node.text.decode()` instead of line-based `_get_line_range()`.

---

## Summary of Changes Required

| File | Lines | Change |
|------|-------|--------|
| `parsers/python_helpers/node_extractors.py` | 47, 154 | `content = node.text.decode()` |
| `parsers/python_helpers/fallback_parser.py` | 57, 86 | `content = node.text.decode()` |
| `parsers/javascript.py` | 145, 186, 226, 265, 295, 460, 487, 515 | `content = node.text.decode()` |
| `parsers/dart.py` | 154, 184, 213, 243, 272, 356, 399, 429, 468 | `content = node.text.decode()` |
| `parsers/php.py` | 176, 216, 246, 280, 314, 348, 458, 493, 526, 573 | `content = node.text.decode()` |
| `parsers/ruby.py` | 171, 209 | `content = node.text.decode()` |

**Total changes**: ~40 locations across 6 files

---

## Next Steps

1. **Implement fix** in all tree-sitter parsers (change `_get_line_range()` to `node.text.decode()`)
2. **Write tests** for minified file chunk_id uniqueness
3. **Validate** no duplicate chunk_ids after fix
4. **Regression test** normal files still work correctly
5. **Update version** and document fix in changelog
