StarPunk/docs/design/v1.0.0/database-migration-conflict-diagnosis.md

Database Migration Conflict Diagnosis Report

Executive Summary

The v1.0.0-rc.2 container is failing because migration 002 attempts to CREATE TABLE authorization_codes, but this table already exists in the production database (created by v1.0.0-rc.1's SCHEMA_SQL).

Issue Details

Error Message

Migration 002_secure_tokens_and_authorization_codes.sql failed: table authorization_codes already exists

Root Cause

Conflicting Database Initialization Strategies

1. SCHEMA_SQL in database.py (lines 58-76): Creates the authorization_codes table directly
2. Migration 002 (line 33): Also attempts to CREATE TABLE authorization_codes

The production database was initialized with v1.0.0-rc.1's SCHEMA_SQL, which created the table. When v1.0.0-rc.2 runs, migration 002 fails because the table already exists.

Database State Analysis

What v1.0.0-rc.1 Created (via SCHEMA_SQL)

-- From database.py lines 58-76
CREATE TABLE IF NOT EXISTS authorization_codes (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    code_hash TEXT UNIQUE NOT NULL,
    me TEXT NOT NULL,
    client_id TEXT NOT NULL,
    redirect_uri TEXT NOT NULL,
    scope TEXT,
    state TEXT,
    code_challenge TEXT,
    code_challenge_method TEXT,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP NOT NULL,
    used_at TIMESTAMP
);

CREATE INDEX IF NOT EXISTS idx_auth_codes_hash ON authorization_codes(code_hash);
CREATE INDEX IF NOT EXISTS idx_auth_codes_expires ON authorization_codes(expires_at);

What Migration 002 Tries to Do

-- From migration 002 lines 33-46
CREATE TABLE authorization_codes (  -- NO "IF NOT EXISTS" clause!
    -- Same structure as above
);

The migration uses CREATE TABLE without IF NOT EXISTS, causing it to fail when the table already exists.

The Good News: System Already Has the Solution

The migrations.py file has sophisticated logic to handle this exact scenario:

Detection Logic (migrations.py lines 176-211)

def is_migration_needed(conn, migration_name):
    if migration_name == "002_secure_tokens_and_authorization_codes.sql":
        # Check if tables exist
        if not table_exists(conn, 'authorization_codes'):
            return True  # Run full migration
        if not column_exists(conn, 'tokens', 'token_hash'):
            return True  # Run full migration

        # Check if indexes exist
        has_all_indexes = (
            index_exists(conn, 'idx_tokens_hash') and
            index_exists(conn, 'idx_tokens_me') and
            # ... other index checks
        )

        if not has_all_indexes:
            # Tables exist but indexes missing
            # Don't run full migration, handle separately
            return False

Resolution Logic (migrations.py lines 383-410)

When tables exist but indexes are missing:

if migration_name == "002_secure_tokens_and_authorization_codes.sql":
    # Create only missing indexes
    indexes_to_create = []
    if not index_exists(conn, 'idx_tokens_hash'):
        indexes_to_create.append("CREATE INDEX idx_tokens_hash ON tokens(token_hash)")
    # ... check and create other indexes

    # Apply indexes without running full migration
    for index_sql in indexes_to_create:
        conn.execute(index_sql)

    # Mark migration as applied
    conn.execute(
        "INSERT INTO schema_migrations (migration_name) VALUES (?)",
        (migration_name,)
    )

Why Is It Still Failing?

The error suggests the smart detection logic isn't being triggered. Possible reasons:

1. Migration Already Marked as Applied: Check if schema_migrations table already has migration 002 listed
2. Different Code Path: The production container might not be using the smart detection path
3. Transaction Rollback: An earlier error might have left the database in an inconsistent state

Immediate Solution

Option 1: Verify Smart Detection Is Working

The system SHOULD handle this automatically. If it's not, check:

1. Is migrations.py line 378 being reached? (migration_count == 0 check)
2. Is is_migration_needed() being called for migration 002?
3. Are the table existence checks working correctly?

Option 2: Manual Database Fix (if smart detection fails)

-- Check current state
SELECT * FROM schema_migrations WHERE migration_name LIKE '%002%';

-- If migration 002 is NOT listed, mark it as applied
INSERT INTO schema_migrations (migration_name)
VALUES ('002_secure_tokens_and_authorization_codes.sql');

-- Ensure indexes exist (if missing)
CREATE INDEX IF NOT EXISTS idx_tokens_hash ON tokens(token_hash);
CREATE INDEX IF NOT EXISTS idx_tokens_me ON tokens(me);
CREATE INDEX IF NOT EXISTS idx_tokens_expires ON tokens(expires_at);
CREATE INDEX IF NOT EXISTS idx_auth_codes_hash ON authorization_codes(code_hash);
CREATE INDEX IF NOT EXISTS idx_auth_codes_expires ON authorization_codes(expires_at);

Long-term Architecture Fix

Current Issue

SCHEMA_SQL and migrations have overlapping responsibilities:

• SCHEMA_SQL creates authorization_codes table (v1.0.0-rc.1+)
• Migration 002 also creates authorization_codes table

Recommended Solution

Already Implemented! The smart detection in migrations.py handles this correctly.

Why It Should Work

1. When database has tables from SCHEMA_SQL but no migration records:

   • is_migration_needed() detects tables exist
   • Returns False to skip full migration
   • Creates only missing indexes
   • Marks migration as applied

2. The system is designed to be self-healing and handle partial schemas

Verification Steps

1. Check Migration Status:

   SELECT * FROM schema_migrations;
   

2. Check Table Existence:

   SELECT name FROM sqlite_master
   WHERE type='table' AND name='authorization_codes';
   

3. Check Index Existence:

   SELECT name FROM sqlite_master
   WHERE type='index' AND name LIKE 'idx_%';
   

4. Check Schema Version Detection:

   • The is_schema_current() function should return False (missing indexes)
   • This should trigger the smart migration path

Conclusion

The architecture already has the correct solution implemented in migrations.py. The smart detection logic should:

1. Detect that authorization_codes table exists
2. Skip the table creation
3. Create only missing indexes
4. Mark migration 002 as applied

If this isn't working, the issue is likely:

• A bug in the detection logic execution path
• The production database already has migration 002 marked as applied (check schema_migrations)
• A transaction rollback leaving the database in an inconsistent state

The system is designed to handle this exact scenario. If it's failing, we need to debug why the smart detection isn't being triggered.