StarPunk/docs/decisions/ADR-034-full-text-search.md

# ADR-034: Full-Text Search with SQLite FTS5

## Status
Proposed

## Context
Users need the ability to search through their notes efficiently. Currently, finding specific content requires manually browsing through notes or using external tools. A built-in search capability is essential for any content management system, especially as the number of notes grows.

Requirements:
- Fast search across all note content
- Support for phrase searching and boolean operators
- Ranking by relevance
- Minimal performance impact on write operations
- No external dependencies (Elasticsearch, Solr, etc.)
- Works with existing SQLite database

## Decision
Implement full-text search using SQLite's FTS5 (Full-Text Search version 5) extension:

1. **FTS5 Virtual Table**: Create a shadow FTS table that indexes note content
2. **Synchronized Updates**: Keep FTS index in sync with note operations
3. **Search Endpoint**: New `/api/search` endpoint for queries
4. **Search UI**: Simple search interface in the web UI
5. **Advanced Operators**: Support FTS5's query syntax for power users

Database schema:
```sql
-- FTS5 virtual table for note content
CREATE VIRTUAL TABLE IF NOT EXISTS notes_fts USING fts5(
    slug UNINDEXED,  -- For result retrieval, not searchable
    title,           -- Note title (first line)
    content,         -- Full markdown content
    tokenize='porter unicode61'  -- Stem words, handle unicode
);

-- Trigger to keep FTS in sync with notes table
CREATE TRIGGER notes_fts_insert AFTER INSERT ON notes
BEGIN
    INSERT INTO notes_fts (rowid, slug, title, content)
    SELECT id, slug, title_from_content(content), content
    FROM notes WHERE id = NEW.id;
END;

-- Similar triggers for UPDATE and DELETE
```

## Rationale
SQLite FTS5 is the optimal choice because:

1. **Native Integration**: Built into SQLite, no external dependencies
2. **Performance**: Highly optimized C implementation
3. **Features**: Rich query syntax (phrases, NEAR, boolean, wildcards)
4. **Ranking**: Built-in BM25 ranking algorithm
5. **Simplicity**: Just another table in our existing database
6. **Maintenance-free**: No separate search service to manage
7. **Size**: Minimal storage overhead (~30% of original text)

Query capabilities:
- Simple terms: `indieweb`
- Phrases: `"static site"`
- Wildcards: `micro*`
- Boolean: `micropub OR websub`
- Exclusions: `indieweb NOT wordpress`
- Field-specific: `title:announcement`

## Consequences
### Positive
- Powerful search with zero external dependencies
- Fast queries even with thousands of notes
- Rich query syntax for power users
- Automatic stemming (search "running" finds "run", "runs")
- Unicode support for international content
- Integrates seamlessly with existing SQLite database

### Negative
- FTS index increases database size by ~30%
- Initial indexing of existing notes required
- Must maintain sync triggers for consistency
- FTS5 requires SQLite 3.9.0+ (2015, widely available)
- Cannot search in encrypted/binary content

### Performance Characteristics
- Index build: ~1ms per note
- Search query: <10ms for 10,000 notes
- Index size: ~30% of indexed text
- Write overhead: ~5% increase in note creation time

## Alternatives Considered

### Alternative 1: Simple LIKE Queries
```sql
SELECT * FROM notes WHERE content LIKE '%search term%'
```
- **Pros**: No setup, works today
- **Cons**: Extremely slow on large datasets, no ranking, no advanced features
- **Rejected because**: Performance degrades quickly with scale

### Alternative 2: External Search Service (Elasticsearch/Meilisearch)
- **Pros**: More features, dedicated search infrastructure
- **Cons**: External dependency, complex setup, overkill for single-user CMS
- **Rejected because**: Violates minimal philosophy, adds operational complexity

### Alternative 3: Client-Side Search (Lunr.js)
- **Pros**: No server changes needed
- **Cons**: Must download all content to browser, doesn't scale
- **Rejected because**: Impractical beyond a few hundred notes

### Alternative 4: Regex/Grep-based Search
- **Pros**: Powerful pattern matching
- **Cons**: Slow, no ranking, must read all files from disk
- **Rejected because**: Poor performance, no relevance ranking

## Implementation Plan

### Phase 1: Database Schema (2 hours)
1. Add FTS5 table creation to migrations
2. Create sync triggers for INSERT/UPDATE/DELETE
3. Build initial index from existing notes
4. Test sync on note operations

### Phase 2: Search API (2 hours)
1. Create `/api/search` endpoint
2. Implement query parser and validation
3. Add result ranking and pagination
4. Return structured results with snippets

### Phase 3: Search UI (1 hour)
1. Add search box to navigation
2. Create search results page
3. Highlight matching terms in results
4. Add search query syntax help

### Phase 4: Testing (1 hour)
1. Test with various query types
2. Benchmark with large datasets
3. Verify sync triggers work correctly
4. Test Unicode and special characters

## API Design

### Search Endpoint
```
GET /api/search?q={query}&limit=20&offset=0

Response:
{
    "query": "indieweb micropub",
    "total": 15,
    "results": [
        {
            "slug": "implementing-micropub",
            "title": "Implementing Micropub",
            "snippet": "...the <mark>IndieWeb</mark> <mark>Micropub</mark> specification...",
            "rank": 2.4,
            "published": true,
            "created_at": "2024-01-15T10:00:00Z"
        }
    ]
}
```

### Query Syntax Examples
- `indieweb` - Find notes containing "indieweb"
- `"static site"` - Exact phrase
- `micro*` - Prefix search
- `title:announcement` - Search in title only
- `micropub OR websub` - Boolean operators
- `indieweb -wordpress` - Exclusion

## Security Considerations
1. Sanitize queries to prevent SQL injection (FTS5 handles this)
2. Rate limit search endpoint to prevent abuse
3. Only search published notes for anonymous users
4. Escape HTML in snippets to prevent XSS

## Migration Strategy
1. Check SQLite version supports FTS5 (3.9.0+)
2. Create FTS table and triggers in migration
3. Build initial index from existing notes
4. Monitor index size and performance
5. Document search syntax for users

## References
- SQLite FTS5 Documentation: https://www.sqlite.org/fts5.html
- BM25 Ranking: https://en.wikipedia.org/wiki/Okapi_BM25
- FTS5 Performance: https://www.sqlite.org/fts5.html#performance