StarPunk/docs/decisions/ADR-033-database-migration-redesign.md

ADR-033: Database Migration System Redesign

Status

Proposed

Context

The current migration system has a critical flaw: duplicate schema definitions exist between SCHEMA_SQL (used for fresh installs) and individual migration files. This violates the DRY principle and creates maintenance burden. When schema changes are made, developers must remember to update both locations, leading to potential inconsistencies.

Current problems:

1. Duplicate schema definitions in SCHEMA_SQL and migration files
2. Risk of schema drift between fresh installs and upgraded databases
3. Maintenance overhead of keeping two schema sources in sync
4. Confusion about which schema definition is authoritative

Decision

Implement an INITIAL_SCHEMA_SQL approach where:

1. Single Source of Truth: The initial schema (v1.0.0 state) is defined once in INITIAL_SCHEMA_SQL
2. Migration-Only Changes: All schema changes after v1.0.0 are defined only in migration files
3. Fresh Install Path: New installations run INITIAL_SCHEMA_SQL + all migrations in sequence
4. Upgrade Path: Existing installations only run new migrations from their current version
5. Version Tracking: The migrations table continues to track applied migrations
6. Lightweight System: Maintain custom migration system without heavyweight ORMs

Implementation approach:

# Conceptual flow (not actual code)
def initialize_database():
    if is_fresh_install():
        execute(INITIAL_SCHEMA_SQL)  # v1.0.0 schema
        mark_initial_version()
    apply_pending_migrations()  # Apply any migrations after v1.0.0

Rationale

This approach provides several benefits:

1. DRY Compliance: Schema for any version is defined exactly once
2. Clear History: Migration files form a clear changelog of schema evolution
3. Reduced Errors: No risk of forgetting to update duplicate definitions
4. Maintainability: Easier to understand what changed when
5. Simplicity: Still lightweight, no heavy dependencies
6. Compatibility: Works with existing migration infrastructure

Alternative approaches considered:

• SQLAlchemy/Alembic: Too heavyweight for a minimal CMS
• Django-style migrations: Requires ORM, adds complexity
• Status quo: Maintaining duplicate schemas is error-prone
• Single evolving schema file: Loses history of changes

Consequences

Positive

• Single source of truth for each schema state
• Clear separation between initial schema and evolution
• Easier onboarding for new developers
• Reduced maintenance burden
• Better documentation of schema evolution

Negative

• One-time migration to new system required
• Must carefully preserve v1.0.0 schema state in INITIAL_SCHEMA_SQL
• Fresh installs run more SQL statements (initial + migrations)

Implementation Requirements

1. Extract current v1.0.0 schema to INITIAL_SCHEMA_SQL
2. Remove schema definitions from existing migration files
3. Update migration runner to handle initial schema
4. Test both fresh install and upgrade paths thoroughly
5. Document the new approach clearly

Alternatives Considered

Alternative 1: SQLAlchemy/Alembic

• Pros: Industry standard, automatic migration generation
• Cons: Heavy dependency, requires ORM adoption, against minimal philosophy
• Rejected because: Overkill for single-table schema

Alternative 2: Single Evolving Schema File

• Pros: Simple, one file to maintain
• Cons: No history, can't track changes, upgrade path unclear
• Rejected because: Loses important schema evolution history

Alternative 3: Status Quo (Duplicate Schemas)

• Pros: Already implemented, works currently
• Cons: DRY violation, error-prone, maintenance burden
• Rejected because: Technical debt will compound over time

Migration Plan

1. Phase 1: Document exact v1.0.0 schema state
2. Phase 2: Create INITIAL_SCHEMA_SQL from current state
3. Phase 3: Refactor migration system to use new approach
4. Phase 4: Test extensively with both paths
5. Phase 5: Deploy in v1.1.0 with clear upgrade instructions

References

• ADR-032: Migration Requirements (parent decision)
• Issue: Database schema duplication
• Similar approach: Rails migrations with schema.rb