StarPunk/docs/reports/v1.1.0-implementation-plan.md

StarPunk v1.1.0 Implementation Plan

Executive Summary

Version 1.1.0 focuses on three high-value features that enhance usability while maintaining our minimal philosophy. This release addresses critical technical debt (migration system), adds essential functionality (search), and improves user control (custom slugs).

Release Overview

• Version: 1.1.0
• Codename: "Searchlight"
• Theme: Enhanced discovery and control
• Estimated Effort: 16-20 hours
• Priority: High (addresses user feedback and technical debt)

Critical Issue: RSS Feed Ordering

Investigation Results

Finding: The RSS feed is already correctly implemented in reverse chronological order (newest first).

Evidence:

• list_notes() function defaults to order_dir="DESC" (descending = newest first)
• SQL query uses ORDER BY created_at DESC
• Feed generation receives notes in correct order

Conclusion: No bug exists. The user's perception may be incorrect, or they may be seeing cached content.

Action Required: None. Document the correct behavior and suggest cache clearing if users report chronological ordering.

Feature Components

1. Database Migration System Redesign (CRITICAL)

Priority: CRITICAL - Must be done first ADR: ADR-033 Effort: 4-6 hours

Problem

• Duplicate schema definitions in SCHEMA_SQL and migration files
• Risk of schema drift between fresh installs and upgrades
• Violates DRY principle

Solution Architecture

# New structure
INITIAL_SCHEMA_SQL = """-- v1.0.0 schema frozen in time"""
migrations = [
    # Only changes after v1.0.0
    "001_add_search_index.sql",
    "002_add_custom_fields.sql"
]

def initialize_database():
    if fresh_install():
        execute(INITIAL_SCHEMA_SQL)
        mark_as_v1_0_0()
    apply_pending_migrations()

Implementation Tasks

1. Extract v1.0.0 Schema (1 hour)

   • Document current production schema
   • Create INITIAL_SCHEMA_SQL constant
   • Verify against existing installations

2. Refactor Migration System (2 hours)

   • Modify migrations.py to use new approach
   • Separate fresh install from upgrade path
   • Update version tracking logic

3. Migration Files Cleanup (1 hour)

   • Remove redundant schema from existing migrations
   • Keep only incremental changes
   • Verify migration sequence

4. Testing (2 hours)

   • Test fresh installation path
   • Test upgrade from v1.0.0
   • Test upgrade from v1.0.1
   • Verify schema consistency

Risks

• Breaking existing installations if not careful
• Must maintain backward compatibility
• Need thorough testing of both paths

2. Full-Text Search with FTS5

Priority: HIGH - Most requested feature ADR: ADR-034 Effort: 6-8 hours

Architecture Design

-- FTS virtual table
CREATE VIRTUAL TABLE notes_fts USING fts5(
    slug UNINDEXED,
    title,
    content,
    tokenize='porter unicode61'
);

-- Sync triggers
CREATE TRIGGER notes_fts_sync_insert ...
CREATE TRIGGER notes_fts_sync_update ...
CREATE TRIGGER notes_fts_sync_delete ...

API Design

@app.route('/api/search')
def search():
    query = request.args.get('q')
    results = db.execute("""
        SELECT slug, snippet(notes_fts, 2, '<mark>', '</mark>', '...', 30)
        FROM notes_fts
        WHERE notes_fts MATCH ?
        ORDER BY rank
        LIMIT 20
    """, [query])
    return jsonify(results)

Implementation Tasks

1. Database Schema (2 hours)

   • Create FTS5 migration
   • Implement sync triggers
   • Build initial index

2. Search API (2 hours)

   • Create /api/search endpoint
   • Implement query validation
   • Add result ranking and snippets
   • Handle pagination

3. Search UI (2 hours)

   • Add search box to navigation
   • Create results page template
   • Implement result highlighting
   • Add query syntax help

4. Testing (2 hours)

   • Test various query types
   • Benchmark performance
   • Verify trigger synchronization
   • Test Unicode content

Performance Targets

• Index building: <1ms per note
• Search latency: <10ms for 10,000 notes
• Index size: ~30% of text size

3. Custom Slugs via Micropub

Priority: MEDIUM - Standards compliance ADR: ADR-035 Effort: 4-5 hours

Design

def create_note_with_slug(content, custom_slug=None):
    if custom_slug:
        slug = sanitize_slug(custom_slug)
        if not validate_slug(slug):
            raise InvalidSlugError()
        if slug_exists(slug):
            slug = make_unique(slug)
    else:
        slug = generate_slug(content)

    return create_note(content, slug=slug)

Validation Rules

• Pattern: ^[a-z0-9]+(?:-[a-z0-9]+)*(?:/[a-z0-9]+(?:-[a-z0-9]+)*)*$
• Max length: 200 characters
• Reserved words: api, admin, auth, feed
• Uniqueness with auto-increment on conflict

Implementation Tasks

1. Core Slug Logic (2 hours)

   • Add slug parameter to note creation
   • Implement validation function
   • Add uniqueness checking
   • Handle conflicts

2. Micropub Integration (1 hour)

   • Extract mp-slug property
   • Pass to note creation
   • Handle validation errors
   • Return proper responses

3. Testing (1.5 hours)

   • Test valid/invalid slugs
   • Test conflict resolution
   • Test with real Micropub clients
   • Test backward compatibility

Security Considerations

• Prevent path traversal (../)
• Block reserved system routes
• Enforce character whitelist
• Normalize case (lowercase only)

Implementation Sequence

Phase 1: Critical Foundation (Week 1)

1. Migration System Redesign (FIRST - blocks everything else)
   • Must be complete before adding new migrations
   • Ensures clean path for FTS5 tables
   • 4-6 hours

Phase 2: Core Features (Week 1-2)

2. Full-Text Search

   • Can begin after migration system ready
   • High user value, most requested
   • 6-8 hours

3. Custom Slugs

   • Can be done in parallel with search
   • Lower complexity, good for end of sprint
   • 4-5 hours

Phase 3: Polish & Release (Week 2)

4. Integration Testing (2 hours)
5. Documentation Updates (1 hour)
6. Release Process (1 hour)

Risk Analysis

High Risks

1. Migration System Breaking Changes

   • Mitigation: Extensive testing, backup instructions
   • Contingency: Rollback procedure documented

2. FTS5 Not Available

   • Mitigation: Check SQLite version in setup
   • Contingency: Graceful degradation

Medium Risks

1. Search Performance Issues

   • Mitigation: Index size monitoring
   • Contingency: Add caching layer

2. Slug Conflicts

   • Mitigation: Auto-increment suffix
   • Contingency: Return clear error messages

Low Risks

1. Increased Database Size
   • Expected: ~30% increase from FTS
   • Acceptable for functionality gained

Success Criteria

Functional Requirements

• Migration system uses single schema source
• Search returns relevant results in <10ms
• Custom slugs accepted via Micropub
• All existing tests pass
• No breaking changes to API

Performance Requirements

• Search latency <10ms for 1000 notes
• Migration completes in <1 second
• No degradation in note creation time

Quality Requirements

• 100% backward compatibility
• No data loss during migration
• Clear error messages for invalid slugs
• Search results properly escaped (XSS prevention)

Testing Strategy

Unit Tests

• Migration path logic
• Slug validation functions
• Search query parsing
• FTS trigger behavior

Integration Tests

• Fresh install flow
• Upgrade from v1.0.0/v1.0.1
• Micropub with custom slugs
• Search API responses

Manual Testing

• Search UI functionality
• Various search queries
• Micropub client compatibility
• Performance benchmarks

Documentation Updates

User Documentation

• Search syntax guide
• Custom slug usage
• Migration instructions

Developer Documentation

• New migration system explanation
• FTS5 implementation details
• Slug validation rules

API Documentation

• /api/search endpoint
• mp-slug property handling
• Error response formats

Release Checklist

Pre-Release

• All tests passing
• Documentation updated
• CHANGELOG.md updated
• Version bumped to 1.1.0
• Migration tested on copy of production

Release

• Tag v1.1.0
• Build container
• Update release notes
• Announce features

Post-Release

• Monitor for issues
• Gather user feedback
• Plan v1.2.0 based on feedback

Estimated Timeline

Week 1 (20-25 hours)

• Day 1-2: Migration system redesign (6h)
• Day 3-4: Full-text search implementation (8h)
• Day 5: Custom slugs implementation (5h)

Week 2 (5-8 hours)

• Day 1: Integration testing (2h)
• Day 2: Documentation and release prep (3h)
• Day 3: Release and monitoring

Total Estimated Effort: 16-20 hours of focused development

Conclusion

Version 1.1.0 represents a significant improvement in usability while maintaining our minimal philosophy. The migration system redesign eliminates technical debt, full-text search adds essential functionality, and custom slugs improve standards compliance.

The implementation should proceed in the order specified, with the migration system being absolutely critical to complete first. Each feature has been designed to be simple, elegant, and maintainable.

Remember our core principle: "Every line of code must justify its existence."