# Qdrant Text Indexing Strategy for Card Building

## Overview

Our card building system stores component metadata in Qdrant. Beyond vector embeddings,
we can leverage Qdrant's full-text indexing features to enable:
- Faster keyword searches
- Fuzzy matching for user input variations
- Phrase matching for multi-word component names
- Language-aware stemming for documentation search

## Current Architecture

Our `mcp_gchat_cards_v7` collection has these payload fields that could benefit:

| Field | Current Type | Content Example | Text Index Candidate? |
|-------|-------------|-----------------|----------------------|
| `name` | keyword | "DecoratedText" | Yes - for component lookup |
| `full_path` | keyword | "card_framework.v2.widgets.DecoratedText" | Yes - for path search |
| `docstring` | text | "A widget that displays..." | **High value** |
| `relationships.nl_descriptions` | text | "decorated text with icon, button..." | **High value** |
| `relationships.child_classes` | list | ["Icon", "Button", "OnClick"] | Array filter works |

## Feature Analysis

### 1. ASCII Folding (v1.16.0+)

**What it does**: Converts Unicode characters to ASCII equivalents (ã→a, é→e)

**Use case for us**:
- Users might search "cafe" when docs contain "café"
- Component descriptions might use special chars: "naïve implementation"

**Where to apply**:
```python
# On docstring and nl_descriptions fields
client.create_payload_index(
    collection_name="mcp_gchat_cards_v7",
    field_name="docstring",
    field_schema=models.TextIndexParams(
        type=models.TextIndexType.TEXT,
        tokenizer=models.TokenizerType.WORD,
        ascii_folding=True,  # Enable
    ),
)
```

**Impact**: Low priority for our use case - component names are ASCII-only.

---

### 2. Stemming (Snowball)

**What it does**: Reduces words to root form (running/runs/runner → "run")

**Use case for us**:
- Search "buttons" matches "Button", "ButtonList"
- Search "selecting" matches "SelectionInput"
- Search "clickable" matches "OnClick"

**Where to apply**:
```python
# On docstring and nl_descriptions for natural language search
client.create_payload_index(
    collection_name="mcp_gchat_cards_v7",
    field_name="relationships.nl_descriptions",
    field_schema=models.TextIndexParams(
        type=models.TextIndexType.TEXT,
        tokenizer=models.TokenizerType.WORD,
        stemmer=models.SnowballParams(
            type=models.Snowball.SNOWBALL,
            language=models.SnowballLanguage.ENGLISH
        )
    ),
)
```

**Impact**: **High value** - Helps match user queries like "add a clickable icon" to
components that have "click" in their relationships.

---

### 3. Stopwords

**What it does**: Filters common words (the, is, at, which, on)

**Use case for us**:
- Ignore "the", "a", "with" in searches
- Focus on meaningful terms: "button with icon" → search "button icon"

**Where to apply**:
```python
client.create_payload_index(
    collection_name="mcp_gchat_cards_v7",
    field_name="docstring",
    field_schema=models.TextIndexParams(
        type=models.TextIndexType.TEXT,
        tokenizer=models.TokenizerType.WORD,
        stopwords=models.StopwordsSet(
            languages=[models.Language.ENGLISH],
            custom=[
                "widget",  # Too generic in our context
                "component",  # Too generic
                "gchat",  # Domain-specific noise
            ]
        ),
    ),
)
```

**Impact**: Medium - Helps with longer docstring searches, reduces noise.

---

### 4. Phrase Search

**What it does**: Matches exact word sequences ("machine learning" as a phrase)

**Use case for us**:
- Match "decorated text" exactly, not "text" AND "decorated" separately
- Match "button list" vs just "button" or "list"
- Match "overflow menu" as exact phrase

**Where to apply**:
```python
client.create_payload_index(
    collection_name="mcp_gchat_cards_v7",
    field_name="name",  # Component names
    field_schema=models.TextIndexParams(
        type=models.TextIndexType.TEXT,
        tokenizer=models.TokenizerType.WORD,
        lowercase=True,
        phrase_matching=True,  # Enable
    ),
)

# Search with phrase:
client.scroll(
    collection_name="mcp_gchat_cards_v7",
    scroll_filter=models.Filter(
        must=[
            models.FieldCondition(
                key="name",
                match=models.MatchText(text='"Decorated Text"')  # Quotes for phrase
            )
        ]
    )
)
```

**Impact**: **High value** - Critical for multi-word component names.

---

## Recommended Index Configuration

```python
from qdrant_client import models

def create_text_indices(client, collection_name: str):
    """Create optimized text indices for card component search."""

    # 1. Component name index - phrase matching for multi-word names
    client.create_payload_index(
        collection_name=collection_name,
        field_name="name",
        field_schema=models.TextIndexParams(
            type=models.TextIndexType.TEXT,
            tokenizer=models.TokenizerType.WORD,
            lowercase=True,
            phrase_matching=True,
        ),
    )

    # 2. Docstring index - stemming + stopwords for NL search
    client.create_payload_index(
        collection_name=collection_name,
        field_name="docstring",
        field_schema=models.TextIndexParams(
            type=models.TextIndexType.TEXT,
            tokenizer=models.TokenizerType.WORD,
            lowercase=True,
            stemmer=models.SnowballParams(
                type=models.Snowball.SNOWBALL,
                language=models.SnowballLanguage.ENGLISH
            ),
            stopwords=models.Language.ENGLISH,
            ascii_folding=True,
        ),
    )

    # 3. Relationship descriptions - stemming for NL matching
    client.create_payload_index(
        collection_name=collection_name,
        field_name="relationships.nl_descriptions",
        field_schema=models.TextIndexParams(
            type=models.TextIndexType.TEXT,
            tokenizer=models.TokenizerType.WORD,
            lowercase=True,
            stemmer=models.SnowballParams(
                type=models.Snowball.SNOWBALL,
                language=models.SnowballLanguage.ENGLISH
            ),
        ),
    )
```

---

## Multi-Module Considerations

For supporting multiple modules (Gmail, Sheets, etc.), we should:

1. **Add module prefix to component names** in payload:
   - `gchat:Button` vs `gmail:Button`
   - Already supported by SymbolGenerator's prefix system

2. **Create separate collections** per module:
   - `mcp_gchat_cards_v7`
   - `mcp_gmail_messages_v7`
   - Keeps indices focused and performant

3. **Or use filtered searches** with a `module` field:
   ```python
   client.scroll(
       scroll_filter=models.Filter(
           must=[
               models.FieldCondition(key="module", match=models.MatchValue(value="gchat")),
               models.FieldCondition(key="name", match=models.MatchText(text="Button")),
           ]
       )
   )
   ```

---

## Implementation Priority

| Feature | Priority | Effort | Benefit |
|---------|----------|--------|---------|
| Phrase matching on `name` | High | Low | Critical for multi-word components |
| Stemming on `nl_descriptions` | High | Low | Better NL query matching |
| Stopwords on `docstring` | Medium | Low | Cleaner searches |
| ASCII folding | Low | Low | Edge case handling |
| Multi-module support | High | Medium | Required for Gmail, Sheets, etc. |

---

## Next Steps

1. Add text index creation to `scripts/initialize_v7_collection.py`
2. Test phrase search for component lookup
3. Benchmark stemmed vs non-stemmed NL queries
4. Plan multi-module collection strategy
