StarPunk/docs/reports/v1.1.0-implementation-plan.md

# StarPunk v1.1.0 Implementation Plan

## Executive Summary
Version 1.1.0 focuses on three high-value features that enhance usability while maintaining our minimal philosophy. This release addresses critical technical debt (migration system), adds essential functionality (search), and improves user control (custom slugs).

## Release Overview
- **Version**: 1.1.0
- **Codename**: "Searchlight"
- **Theme**: Enhanced discovery and control
- **Estimated Effort**: 16-20 hours
- **Priority**: High (addresses user feedback and technical debt)

## Critical Issue: RSS Feed Ordering

### Investigation Results
**Finding**: The RSS feed is already correctly implemented in reverse chronological order (newest first).

**Evidence**:
- `list_notes()` function defaults to `order_dir="DESC"` (descending = newest first)
- SQL query uses `ORDER BY created_at DESC`
- Feed generation receives notes in correct order

**Conclusion**: No bug exists. The user's perception may be incorrect, or they may be seeing cached content.

**Action Required**: None. Document the correct behavior and suggest cache clearing if users report chronological ordering.

## Feature Components

### 1. Database Migration System Redesign (CRITICAL)
**Priority**: CRITICAL - Must be done first
**ADR**: ADR-033
**Effort**: 4-6 hours

#### Problem
- Duplicate schema definitions in SCHEMA_SQL and migration files
- Risk of schema drift between fresh installs and upgrades
- Violates DRY principle

#### Solution Architecture
```python
# New structure
INITIAL_SCHEMA_SQL = """-- v1.0.0 schema frozen in time"""
migrations = [
    # Only changes after v1.0.0
    "001_add_search_index.sql",
    "002_add_custom_fields.sql"
]

def initialize_database():
    if fresh_install():
        execute(INITIAL_SCHEMA_SQL)
        mark_as_v1_0_0()
    apply_pending_migrations()
```

#### Implementation Tasks
1. **Extract v1.0.0 Schema** (1 hour)
   - Document current production schema
   - Create INITIAL_SCHEMA_SQL constant
   - Verify against existing installations

2. **Refactor Migration System** (2 hours)
   - Modify `migrations.py` to use new approach
   - Separate fresh install from upgrade path
   - Update version tracking logic

3. **Migration Files Cleanup** (1 hour)
   - Remove redundant schema from existing migrations
   - Keep only incremental changes
   - Verify migration sequence

4. **Testing** (2 hours)
   - Test fresh installation path
   - Test upgrade from v1.0.0
   - Test upgrade from v1.0.1
   - Verify schema consistency

#### Risks
- Breaking existing installations if not careful
- Must maintain backward compatibility
- Need thorough testing of both paths

### 2. Full-Text Search with FTS5
**Priority**: HIGH - Most requested feature
**ADR**: ADR-034
**Effort**: 6-8 hours

#### Architecture Design
```sql
-- FTS virtual table
CREATE VIRTUAL TABLE notes_fts USING fts5(
    slug UNINDEXED,
    title,
    content,
    tokenize='porter unicode61'
);

-- Sync triggers
CREATE TRIGGER notes_fts_sync_insert ...
CREATE TRIGGER notes_fts_sync_update ...
CREATE TRIGGER notes_fts_sync_delete ...
```

#### API Design
```python
@app.route('/api/search')
def search():
    query = request.args.get('q')
    results = db.execute("""
        SELECT slug, snippet(notes_fts, 2, '<mark>', '</mark>', '...', 30)
        FROM notes_fts
        WHERE notes_fts MATCH ?
        ORDER BY rank
        LIMIT 20
    """, [query])
    return jsonify(results)
```

#### Implementation Tasks
1. **Database Schema** (2 hours)
   - Create FTS5 migration
   - Implement sync triggers
   - Build initial index

2. **Search API** (2 hours)
   - Create `/api/search` endpoint
   - Implement query validation
   - Add result ranking and snippets
   - Handle pagination

3. **Search UI** (2 hours)
   - Add search box to navigation
   - Create results page template
   - Implement result highlighting
   - Add query syntax help

4. **Testing** (2 hours)
   - Test various query types
   - Benchmark performance
   - Verify trigger synchronization
   - Test Unicode content

#### Performance Targets
- Index building: <1ms per note
- Search latency: <10ms for 10,000 notes
- Index size: ~30% of text size

### 3. Custom Slugs via Micropub
**Priority**: MEDIUM - Standards compliance
**ADR**: ADR-035
**Effort**: 4-5 hours

#### Design
```python
def create_note_with_slug(content, custom_slug=None):
    if custom_slug:
        slug = sanitize_slug(custom_slug)
        if not validate_slug(slug):
            raise InvalidSlugError()
        if slug_exists(slug):
            slug = make_unique(slug)
    else:
        slug = generate_slug(content)

    return create_note(content, slug=slug)
```

#### Validation Rules
- Pattern: `^[a-z0-9]+(?:-[a-z0-9]+)*(?:/[a-z0-9]+(?:-[a-z0-9]+)*)*$`
- Max length: 200 characters
- Reserved words: `api`, `admin`, `auth`, `feed`
- Uniqueness with auto-increment on conflict

#### Implementation Tasks
1. **Core Slug Logic** (2 hours)
   - Add slug parameter to note creation
   - Implement validation function
   - Add uniqueness checking
   - Handle conflicts

2. **Micropub Integration** (1 hour)
   - Extract `mp-slug` property
   - Pass to note creation
   - Handle validation errors
   - Return proper responses

3. **Testing** (1.5 hours)
   - Test valid/invalid slugs
   - Test conflict resolution
   - Test with real Micropub clients
   - Test backward compatibility

#### Security Considerations
- Prevent path traversal (`../`)
- Block reserved system routes
- Enforce character whitelist
- Normalize case (lowercase only)

## Implementation Sequence

### Phase 1: Critical Foundation (Week 1)
1. **Migration System Redesign** (FIRST - blocks everything else)
   - Must be complete before adding new migrations
   - Ensures clean path for FTS5 tables
   - 4-6 hours

### Phase 2: Core Features (Week 1-2)
2. **Full-Text Search**
   - Can begin after migration system ready
   - High user value, most requested
   - 6-8 hours

3. **Custom Slugs**
   - Can be done in parallel with search
   - Lower complexity, good for end of sprint
   - 4-5 hours

### Phase 3: Polish & Release (Week 2)
4. **Integration Testing** (2 hours)
5. **Documentation Updates** (1 hour)
6. **Release Process** (1 hour)

## Risk Analysis

### High Risks
1. **Migration System Breaking Changes**
   - Mitigation: Extensive testing, backup instructions
   - Contingency: Rollback procedure documented

2. **FTS5 Not Available**
   - Mitigation: Check SQLite version in setup
   - Contingency: Graceful degradation

### Medium Risks
1. **Search Performance Issues**
   - Mitigation: Index size monitoring
   - Contingency: Add caching layer

2. **Slug Conflicts**
   - Mitigation: Auto-increment suffix
   - Contingency: Return clear error messages

### Low Risks
1. **Increased Database Size**
   - Expected: ~30% increase from FTS
   - Acceptable for functionality gained

## Success Criteria

### Functional Requirements
- [ ] Migration system uses single schema source
- [ ] Search returns relevant results in <10ms
- [ ] Custom slugs accepted via Micropub
- [ ] All existing tests pass
- [ ] No breaking changes to API

### Performance Requirements
- [ ] Search latency <10ms for 1000 notes
- [ ] Migration completes in <1 second
- [ ] No degradation in note creation time

### Quality Requirements
- [ ] 100% backward compatibility
- [ ] No data loss during migration
- [ ] Clear error messages for invalid slugs
- [ ] Search results properly escaped (XSS prevention)

## Testing Strategy

### Unit Tests
- Migration path logic
- Slug validation functions
- Search query parsing
- FTS trigger behavior

### Integration Tests
- Fresh install flow
- Upgrade from v1.0.0/v1.0.1
- Micropub with custom slugs
- Search API responses

### Manual Testing
- Search UI functionality
- Various search queries
- Micropub client compatibility
- Performance benchmarks

## Documentation Updates

### User Documentation
- Search syntax guide
- Custom slug usage
- Migration instructions

### Developer Documentation
- New migration system explanation
- FTS5 implementation details
- Slug validation rules

### API Documentation
- `/api/search` endpoint
- `mp-slug` property handling
- Error response formats

## Release Checklist

### Pre-Release
- [ ] All tests passing
- [ ] Documentation updated
- [ ] CHANGELOG.md updated
- [ ] Version bumped to 1.1.0
- [ ] Migration tested on copy of production

### Release
- [ ] Tag v1.1.0
- [ ] Build container
- [ ] Update release notes
- [ ] Announce features

### Post-Release
- [ ] Monitor for issues
- [ ] Gather user feedback
- [ ] Plan v1.2.0 based on feedback

## Estimated Timeline

### Week 1 (20-25 hours)
- Day 1-2: Migration system redesign (6h)
- Day 3-4: Full-text search implementation (8h)
- Day 5: Custom slugs implementation (5h)

### Week 2 (5-8 hours)
- Day 1: Integration testing (2h)
- Day 2: Documentation and release prep (3h)
- Day 3: Release and monitoring

**Total Estimated Effort**: 16-20 hours of focused development

## Conclusion

Version 1.1.0 represents a significant improvement in usability while maintaining our minimal philosophy. The migration system redesign eliminates technical debt, full-text search adds essential functionality, and custom slugs improve standards compliance.

The implementation should proceed in the order specified, with the migration system being absolutely critical to complete first. Each feature has been designed to be simple, elegant, and maintainable.

Remember our core principle: "Every line of code must justify its existence."