StarPunk/docs/decisions/ADR-031-database-migration-system-redesign.md

ADR-031: Database Migration System Redesign

Status

Proposed

Context

The v1.0.0-rc.1 release exposed a critical flaw in our database initialization and migration system. The system fails when upgrading existing production databases because:

1. SCHEMA_SQL represents the current (latest) schema structure
2. SCHEMA_SQL is executed BEFORE migrations run
3. Existing databases have old table structures that conflict with SCHEMA_SQL's expectations
4. The system tries to create indexes on columns that don't exist yet

This creates an impossible situation where:

• Fresh databases work fine (SCHEMA_SQL creates the latest structure)
• Existing databases fail (SCHEMA_SQL conflicts with old structure)

Decision

Redesign the database initialization system to follow these principles:

1. SCHEMA_SQL represents the initial v0.1.0 schema, not the current schema
2. All schema evolution happens through migrations
3. Migrations run BEFORE schema creation attempts
4. Fresh databases get the initial schema then run ALL migrations

Implementation Strategy

Phase 1: Immediate Fix (v1.0.1)

Remove problematic index creation from SCHEMA_SQL since migrations create them:

# Remove from SCHEMA_SQL:
# CREATE INDEX IF NOT EXISTS idx_tokens_hash ON tokens(token_hash);
# Let migration 002 handle this

Phase 2: Proper Redesign (v1.1.0)

1. Create INITIAL_SCHEMA_SQL with the v0.1.0 database structure

2. Modify init_db() logic:

   def init_db(app=None):
       # 1. Check if database exists and has tables
       if database_exists_with_tables():
           # Existing database - only run migrations
           run_migrations()
       else:
           # Fresh database - create initial schema then migrate
           conn.executescript(INITIAL_SCHEMA_SQL)
           run_all_migrations()
   

3. Add explicit schema versioning:

   CREATE TABLE schema_info (
       version TEXT PRIMARY KEY,
       upgraded_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
   );
   

Rationale

Why Initial Schema + Migrations?

1. Predictable upgrade path: Every database follows the same evolution
2. Testable: Can test upgrades from any version to any version
3. Auditable: Migration history shows exact evolution path
4. Reversible: Can potentially support rollbacks
5. Industry standard: Follows patterns from Rails, Django, Alembic

Why Current Approach Failed

1. Dual source of truth: Schema defined in both SCHEMA_SQL and migrations
2. Temporal coupling: SCHEMA_SQL assumes post-migration state
3. No upgrade path: Can't get from old state to new state
4. Hidden dependencies: Index creation depends on migration execution

Consequences

Positive

• Reliable database upgrades from any version
• Clear separation of concerns (initial vs evolution)
• Easier to test migration paths
• Follows established patterns
• Supports future rollback capabilities

Negative

• Requires maintaining historical schema (INITIAL_SCHEMA_SQL)
• Fresh databases take longer to initialize (run all migrations)
• More complex initialization logic
• Need to reconstruct v0.1.0 schema

Migration Path

1. v1.0.1: Quick fix - remove conflicting indexes from SCHEMA_SQL
2. v1.0.1: Add manual upgrade instructions for production
3. v1.1.0: Implement full redesign with INITIAL_SCHEMA_SQL
4. v1.1.0: Add comprehensive migration testing

Alternatives Considered

1. Dynamic Schema Detection

Approach: Detect existing table structure and conditionally apply indexes

Rejected because:

• Complex conditional logic
• Fragile heuristics
• Doesn't solve root cause
• Hard to test all paths

2. Schema Snapshots

Approach: Maintain schema snapshots for each version, apply appropriate one

Rejected because:

• Maintenance burden
• Storage overhead
• Complex version detection
• Still doesn't provide upgrade path

3. Migration-Only Schema

Approach: No SCHEMA_SQL at all, everything through migrations

Rejected because:

• Slower fresh installations
• Need to maintain migration 000 as "initial schema"
• Harder to see current schema structure
• Goes against SQLite's lightweight philosophy

References

• Rails Database Migrations
• Django Migrations
• Alembic Documentation
• Production incident: v1.0.0-rc.1 deployment failure
• /docs/reports/migration-failure-diagnosis-v1.0.0-rc.1.md

Implementation Checklist

• Create INITIAL_SCHEMA_SQL from v0.1.0 structure
• Modify init_db() to check database state
• Update migration runner to handle fresh databases
• Add schema_info table for version tracking
• Create migration test suite
• Document upgrade procedures
• Test upgrade paths from all released versions