fix: Resolve database migration failure on existing databases

Fixes critical issue where migration 002 indexes already existed in SCHEMA_SQL, causing 'index already exists' errors on databases created before v1.0.0-rc.1. Changes: - Removed duplicate index definitions from SCHEMA_SQL (database.py) - Enhanced migration system to detect and handle indexes properly - Added comprehensive documentation of the fix Version bumped to 1.0.0-rc.2 with full changelog entry. Refs: docs/reports/2025-11-24-migration-fix-v1.0.0-rc.2.md
2025-11-24 13:11:14 -07:00
parent 89758fd1a5
commit 3ed77fd45f
16 changed files with 2723 additions and 18 deletions
--- a/docs/projectplan/v1.1/priority-work.md
+++ b/docs/projectplan/v1.1/priority-work.md
@@ -0,0 +1,218 @@
+# StarPunk v1.1.0: Priority Work Items
+
+## Overview
+
+This document identifies HIGH PRIORITY work items that MUST be completed for the v1.1.0 release. These items address critical issues discovered in production and architectural improvements required for system stability.
+
+**Target Release**: v1.1.0
+**Status**: Planning
+**Created**: 2025-11-24
+
+## Critical Priority Items
+
+These items MUST be completed before v1.1.0 release.
+
+---
+
+### 1. Database Migration System Redesign - Phase 2
+
+**Priority**: CRITICAL
+**ADR**: ADR-032
+**Estimated Effort**: 4-6 hours
+**Dependencies**: None
+**Risk**: Low (backward compatible)
+
+#### Problem
+The current database initialization system fails when upgrading existing production databases because SCHEMA_SQL represents the current schema rather than the initial v0.1.0 baseline. This causes indexes to be created on columns that don't exist yet.
+
+#### Solution
+Implement INITIAL_SCHEMA_SQL as designed in ADR-032 to represent the v0.1.0 baseline schema. All schema evolution will happen through migrations.
+
+#### Implementation Tasks
+
+1. **Create INITIAL_SCHEMA_SQL constant** (`database.py`)
+   ```python
+   INITIAL_SCHEMA_SQL = """
+   -- V0.1.0 baseline schema from commit a68fd57
+   -- [Full SQL as documented in ADR-032]
+   """
+   ```
+
+2. **Modify init_db() function** (`database.py`)
+   - Add database existence check
+   - Use INITIAL_SCHEMA_SQL for fresh databases
+   - Run migrations for all databases
+   - See ADR-032 for complete logic
+
+3. **Add helper functions** (`database.py`)
+   - `database_exists_with_tables()`: Check if database has existing tables
+   - Update imports and error handling
+
+4. **Update existing SCHEMA_SQL** (`database.py`)
+   - Rename to CURRENT_SCHEMA_SQL
+   - Mark as documentation-only (not used for initialization)
+   - Add clear comments explaining purpose
+
+#### Testing Requirements
+
+- [ ] Test fresh database initialization (should create v0.1.0 schema then migrate)
+- [ ] Test upgrade from existing v1.0.0-rc.2 database
+- [ ] Test upgrade from v0.x.x databases if available
+- [ ] Verify all indexes created correctly
+- [ ] Verify no duplicate table/index errors
+- [ ] Test migration tracking (schema_migrations table)
+- [ ] Performance test for fresh install (all migrations)
+
+#### Documentation Updates
+
+- [ ] Update database.py docstrings
+- [ ] Add inline comments explaining dual schema constants
+- [ ] Update deployment documentation
+- [ ] Add production upgrade guide
+- [ ] Update CHANGELOG.md
+
+#### Success Criteria
+
+- Existing databases upgrade without errors
+- Fresh databases initialize correctly
+- All migrations run in proper order
+- No index creation errors
+- Clear upgrade path from any version
+
+---
+
+### 2. IndieAuth Provider Strategy Implementation
+
+**Priority**: HIGH
+**ADR**: ADR-021 (if exists)
+**Estimated Effort**: 8-10 hours
+**Dependencies**: Database migration system working correctly
+**Risk**: Medium (external service dependencies)
+
+#### Problem
+Current IndieAuth implementation may need updates based on production usage patterns and compliance requirements.
+
+#### Implementation Notes
+- Review existing ADR-021-indieauth-provider-strategy.md
+- Implement any pending IndieAuth improvements
+- Ensure full spec compliance
+
+---
+
+## Medium Priority Items
+
+These items SHOULD be completed for v1.1.0 if time permits.
+
+### 3. Full-Text Search Implementation
+
+**Priority**: MEDIUM
+**Reference**: v1.1/potential-features.md
+**Estimated Effort**: 3-4 hours
+**Dependencies**: None
+**Risk**: Low
+
+#### Implementation Approach
+- Use SQLite FTS5 extension
+- Create shadow FTS table for note content
+- Update on note create/update/delete
+- Add search_notes() function to notes.py
+
+---
+
+### 4. Migration System Testing Suite
+
+**Priority**: MEDIUM
+**Estimated Effort**: 4-5 hours
+**Dependencies**: Item #1 (Migration redesign)
+**Risk**: Low
+
+#### Test Coverage Needed
+- Migration ordering tests
+- Rollback simulation tests
+- Schema evolution tests
+- Performance benchmarks
+- CI/CD integration
+
+---
+
+## Implementation Order
+
+1. **First**: Complete Database Migration System Redesign (Critical)
+2. **Second**: Add comprehensive migration tests
+3. **Third**: IndieAuth improvements (if needed)
+4. **Fourth**: Full-text search (if time permits)
+
+## Release Checklist
+
+Before releasing v1.1.0:
+
+- [ ] All CRITICAL items complete
+- [ ] All tests passing
+- [ ] Documentation updated
+- [ ] CHANGELOG.md updated with all changes
+- [ ] Version bumped to 1.1.0
+- [ ] Migration guide written for production systems
+- [ ] Release notes prepared
+- [ ] Docker image tested with migrations
+
+## Risk Mitigation
+
+### Migration System Risks
+- **Risk**: Breaking existing databases
+- **Mitigation**: Comprehensive testing, backward compatibility, clear rollback procedures
+
+### Performance Risks
+- **Risk**: Slow fresh installations (running all migrations)
+- **Mitigation**: Migration performance testing, potential migration squashing in future
+
+### Deployment Risks
+- **Risk**: Production upgrade failures
+- **Mitigation**: Detailed upgrade guide, test on staging first, backup procedures
+
+## Notes for Implementation
+
+### For the Developer Implementing Item #1
+
+1. **Start with ADR-032** for complete design details
+2. **Check git history** for original schema (commit a68fd57)
+3. **Test thoroughly** - this is critical infrastructure
+4. **Consider edge cases**:
+   - Empty database
+   - Partially migrated database
+   - Corrupted migration tracking
+   - Missing migration files
+
+### Key Files to Modify
+
+1. `/home/phil/Projects/starpunk/starpunk/database.py`
+   - Add INITIAL_SCHEMA_SQL constant
+   - Modify init_db() function
+   - Add helper functions
+
+2. `/home/phil/Projects/starpunk/tests/test_migrations.py`
+   - Add new test cases for initial schema
+   - Test upgrade paths
+
+3. `/home/phil/Projects/starpunk/docs/architecture/database.md`
+   - Document schema evolution strategy
+   - Explain dual schema constants
+
+## Success Metrics
+
+- Zero database upgrade failures in production
+- Fresh installation time < 1 second
+- All tests passing
+- Clear documentation for future maintainers
+- Positive user feedback on stability
+
+## References
+
+- [ADR-031: Database Migration System Redesign](/home/phil/Projects/starpunk/docs/decisions/ADR-031-database-migration-system-redesign.md)
+- [ADR-032: Initial Schema SQL Implementation](/home/phil/Projects/starpunk/docs/decisions/ADR-032-initial-schema-sql-implementation.md)
+- [v1.1 Potential Features](/home/phil/Projects/starpunk/docs/projectplan/v1.1/potential-features.md)
+- [Migration Implementation Reports](/home/phil/Projects/starpunk/docs/reports/)
+
+---
+
+*Last Updated: 2025-11-24*
+*Version: 1.0.0-rc.2 → 1.1.0 (planned)*