fix: Resolve database migration failure on existing databases
Fixes critical issue where migration 002 indexes already existed in SCHEMA_SQL, causing 'index already exists' errors on databases created before v1.0.0-rc.1. Changes: - Removed duplicate index definitions from SCHEMA_SQL (database.py) - Enhanced migration system to detect and handle indexes properly - Added comprehensive documentation of the fix Version bumped to 1.0.0-rc.2 with full changelog entry. Refs: docs/reports/2025-11-24-migration-fix-v1.0.0-rc.2.md
This commit is contained in:
393
docs/design/initial-schema-implementation-guide.md
Normal file
393
docs/design/initial-schema-implementation-guide.md
Normal file
@@ -0,0 +1,393 @@
|
||||
# Initial Schema SQL Implementation Guide
|
||||
|
||||
## Overview
|
||||
|
||||
This guide provides step-by-step instructions for implementing the INITIAL_SCHEMA_SQL constant and updating the database initialization system as specified in ADR-032.
|
||||
|
||||
**Priority**: CRITICAL for v1.1.0
|
||||
**Estimated Time**: 4-6 hours
|
||||
**Risk Level**: Low (backward compatible)
|
||||
|
||||
## Pre-Implementation Checklist
|
||||
|
||||
- [ ] Read ADR-031 (Database Migration System Redesign)
|
||||
- [ ] Read ADR-032 (Initial Schema SQL Implementation)
|
||||
- [ ] Review current migrations in `/migrations/` directory
|
||||
- [ ] Backup any test databases
|
||||
- [ ] Ensure test environment is ready
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
### Step 1: Add INITIAL_SCHEMA_SQL Constant
|
||||
|
||||
**File**: `/home/phil/Projects/starpunk/starpunk/database.py`
|
||||
|
||||
**Action**: Add the following constant ABOVE the current SCHEMA_SQL:
|
||||
|
||||
```python
|
||||
# Database schema - V0.1.0 baseline (see ADR-032)
|
||||
# This represents the initial database structure from commit a68fd57
|
||||
# All schema evolution happens through migrations from this baseline
|
||||
INITIAL_SCHEMA_SQL = """
|
||||
-- Notes metadata (content is in files)
|
||||
CREATE TABLE IF NOT EXISTS notes (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
slug TEXT UNIQUE NOT NULL,
|
||||
file_path TEXT UNIQUE NOT NULL,
|
||||
published BOOLEAN DEFAULT 0,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
deleted_at TIMESTAMP,
|
||||
content_hash TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_notes_created_at ON notes(created_at DESC);
|
||||
CREATE INDEX IF NOT EXISTS idx_notes_published ON notes(published);
|
||||
CREATE INDEX IF NOT EXISTS idx_notes_slug ON notes(slug);
|
||||
CREATE INDEX IF NOT EXISTS idx_notes_deleted_at ON notes(deleted_at);
|
||||
|
||||
-- Authentication sessions (IndieLogin)
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
session_token TEXT UNIQUE NOT NULL,
|
||||
me TEXT NOT NULL,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at TIMESTAMP NOT NULL,
|
||||
last_used_at TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_token ON sessions(session_token);
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_expires ON sessions(expires_at);
|
||||
|
||||
-- Micropub access tokens (original insecure version)
|
||||
CREATE TABLE IF NOT EXISTS tokens (
|
||||
token TEXT PRIMARY KEY,
|
||||
me TEXT NOT NULL,
|
||||
client_id TEXT,
|
||||
scope TEXT,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at TIMESTAMP
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_tokens_me ON tokens(me);
|
||||
|
||||
-- CSRF state tokens (for IndieAuth flow)
|
||||
CREATE TABLE IF NOT EXISTS auth_state (
|
||||
state TEXT PRIMARY KEY,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at TIMESTAMP NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_auth_state_expires ON auth_state(expires_at);
|
||||
"""
|
||||
```
|
||||
|
||||
### Step 2: Rename Current SCHEMA_SQL
|
||||
|
||||
**File**: `/home/phil/Projects/starpunk/starpunk/database.py`
|
||||
|
||||
**Action**: Rename the existing SCHEMA_SQL constant and add documentation:
|
||||
|
||||
```python
|
||||
# Current database schema - FOR DOCUMENTATION ONLY
|
||||
# This shows the current complete schema after all migrations
|
||||
# NOT used for database initialization - see INITIAL_SCHEMA_SQL
|
||||
# Updated by migrations 001 and 002
|
||||
CURRENT_SCHEMA_SQL = """
|
||||
[existing SCHEMA_SQL content]
|
||||
"""
|
||||
```
|
||||
|
||||
### Step 3: Add Helper Function
|
||||
|
||||
**File**: `/home/phil/Projects/starpunk/starpunk/database.py`
|
||||
|
||||
**Action**: Add this function before init_db():
|
||||
|
||||
```python
|
||||
def database_exists_with_tables(db_path):
|
||||
"""
|
||||
Check if database exists and has tables
|
||||
|
||||
Args:
|
||||
db_path: Path to SQLite database file
|
||||
|
||||
Returns:
|
||||
bool: True if database exists with at least one table
|
||||
"""
|
||||
import os
|
||||
|
||||
# Check if file exists
|
||||
if not os.path.exists(db_path):
|
||||
return False
|
||||
|
||||
# Check if it has tables
|
||||
try:
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.execute(
|
||||
"SELECT COUNT(*) FROM sqlite_master WHERE type='table'"
|
||||
)
|
||||
table_count = cursor.fetchone()[0]
|
||||
conn.close()
|
||||
return table_count > 0
|
||||
except Exception:
|
||||
return False
|
||||
```
|
||||
|
||||
### Step 4: Update init_db() Function
|
||||
|
||||
**File**: `/home/phil/Projects/starpunk/starpunk/database.py`
|
||||
|
||||
**Action**: Replace the init_db() function with:
|
||||
|
||||
```python
|
||||
def init_db(app=None):
|
||||
"""
|
||||
Initialize database schema and run migrations
|
||||
|
||||
For fresh databases:
|
||||
1. Creates v0.1.0 baseline schema (INITIAL_SCHEMA_SQL)
|
||||
2. Runs all migrations to bring to current version
|
||||
|
||||
For existing databases:
|
||||
1. Skips schema creation (tables already exist)
|
||||
2. Runs only pending migrations
|
||||
|
||||
Args:
|
||||
app: Flask application instance (optional, for config access)
|
||||
"""
|
||||
if app:
|
||||
db_path = app.config["DATABASE_PATH"]
|
||||
logger = app.logger
|
||||
else:
|
||||
# Fallback to default path
|
||||
db_path = Path("./data/starpunk.db")
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Ensure parent directory exists
|
||||
db_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Check if this is an existing database
|
||||
if database_exists_with_tables(db_path):
|
||||
# Existing database - skip schema creation, only run migrations
|
||||
logger.info(f"Existing database found: {db_path}")
|
||||
logger.info("Running pending migrations...")
|
||||
else:
|
||||
# Fresh database - create initial v0.1.0 schema
|
||||
logger.info(f"Creating new database: {db_path}")
|
||||
conn = sqlite3.connect(db_path)
|
||||
try:
|
||||
# Create v0.1.0 baseline schema
|
||||
conn.executescript(INITIAL_SCHEMA_SQL)
|
||||
conn.commit()
|
||||
logger.info("Created initial v0.1.0 database schema")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create initial schema: {e}")
|
||||
raise
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
# Run migrations (for both fresh and existing databases)
|
||||
# This will apply ALL migrations for fresh databases,
|
||||
# or only pending migrations for existing databases
|
||||
from starpunk.migrations import run_migrations
|
||||
|
||||
try:
|
||||
run_migrations(db_path, logger)
|
||||
except Exception as e:
|
||||
logger.error(f"Migration failed: {e}")
|
||||
raise
|
||||
```
|
||||
|
||||
### Step 5: Update Tests
|
||||
|
||||
**File**: `/home/phil/Projects/starpunk/tests/test_migrations.py`
|
||||
|
||||
**Add these test cases**:
|
||||
|
||||
```python
|
||||
def test_fresh_database_initialization(tmp_path):
|
||||
"""Test that fresh database gets initial schema then migrations"""
|
||||
db_path = tmp_path / "test.db"
|
||||
|
||||
# Initialize fresh database
|
||||
init_db_with_path(db_path)
|
||||
|
||||
# Verify initial tables exist
|
||||
conn = sqlite3.connect(db_path)
|
||||
cursor = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
|
||||
)
|
||||
tables = [row[0] for row in cursor.fetchall()]
|
||||
|
||||
# Should have all tables including migration tracking
|
||||
assert "notes" in tables
|
||||
assert "sessions" in tables
|
||||
assert "tokens" in tables
|
||||
assert "auth_state" in tables
|
||||
assert "schema_migrations" in tables
|
||||
assert "authorization_codes" in tables # Added by migration 002
|
||||
|
||||
# Verify migrations were applied
|
||||
cursor = conn.execute("SELECT COUNT(*) FROM schema_migrations")
|
||||
migration_count = cursor.fetchone()[0]
|
||||
assert migration_count >= 2 # At least migrations 001 and 002
|
||||
|
||||
conn.close()
|
||||
|
||||
|
||||
def test_existing_database_upgrade(tmp_path):
|
||||
"""Test that existing database only runs pending migrations"""
|
||||
db_path = tmp_path / "test.db"
|
||||
|
||||
# Create a database with v0.1.0 schema manually
|
||||
conn = sqlite3.connect(db_path)
|
||||
conn.executescript(INITIAL_SCHEMA_SQL)
|
||||
conn.commit()
|
||||
conn.close()
|
||||
|
||||
# Run init_db on existing database
|
||||
init_db_with_path(db_path)
|
||||
|
||||
# Verify migrations were applied
|
||||
conn = sqlite3.connect(db_path)
|
||||
|
||||
# Check that migration 001 was applied (code_verifier column)
|
||||
cursor = conn.execute("PRAGMA table_info(auth_state)")
|
||||
columns = [row[1] for row in cursor.fetchall()]
|
||||
assert "code_verifier" in columns
|
||||
|
||||
# Check that migration 002 was applied (authorization_codes table)
|
||||
cursor = conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name='authorization_codes'"
|
||||
)
|
||||
assert cursor.fetchone() is not None
|
||||
|
||||
conn.close()
|
||||
```
|
||||
|
||||
### Step 6: Manual Testing Procedure
|
||||
|
||||
1. **Test Fresh Database**:
|
||||
```bash
|
||||
# Backup existing database
|
||||
mv data/starpunk.db data/starpunk.db.backup
|
||||
|
||||
# Start application (will create fresh database)
|
||||
uv run python app.py
|
||||
|
||||
# Verify application starts without errors
|
||||
# Check logs for "Created initial v0.1.0 database schema"
|
||||
# Check logs for "Applied migration: 001_add_code_verifier_to_auth_state.sql"
|
||||
# Check logs for "Applied migration: 002_secure_tokens_and_authorization_codes.sql"
|
||||
```
|
||||
|
||||
2. **Test Existing Database**:
|
||||
```bash
|
||||
# Restore backup
|
||||
cp data/starpunk.db.backup data/starpunk.db
|
||||
|
||||
# Start application
|
||||
uv run python app.py
|
||||
|
||||
# Verify application starts without errors
|
||||
# Check logs for "Existing database found"
|
||||
# Check logs for migration status
|
||||
```
|
||||
|
||||
3. **Test Database Queries**:
|
||||
```bash
|
||||
sqlite3 data/starpunk.db
|
||||
|
||||
# Check tables
|
||||
.tables
|
||||
|
||||
# Check schema_migrations
|
||||
SELECT * FROM schema_migrations;
|
||||
|
||||
# Verify authorization_codes table exists
|
||||
.schema authorization_codes
|
||||
|
||||
# Verify tokens table has token_hash column
|
||||
.schema tokens
|
||||
```
|
||||
|
||||
### Step 7: Update Documentation
|
||||
|
||||
**File**: `/home/phil/Projects/starpunk/docs/architecture/database.md`
|
||||
|
||||
**Add section**:
|
||||
|
||||
```markdown
|
||||
## Schema Evolution Strategy
|
||||
|
||||
StarPunk uses a baseline + migrations approach for schema management:
|
||||
|
||||
1. **INITIAL_SCHEMA_SQL**: Represents the v0.1.0 baseline schema
|
||||
2. **Migrations**: All schema changes applied sequentially
|
||||
3. **CURRENT_SCHEMA_SQL**: Documentation of current complete schema
|
||||
|
||||
This ensures:
|
||||
- Predictable upgrade paths from any version
|
||||
- Clear schema history through migrations
|
||||
- Testable database evolution
|
||||
```
|
||||
|
||||
## Validation Checklist
|
||||
|
||||
After implementation, verify:
|
||||
|
||||
- [ ] Fresh database initialization works
|
||||
- [ ] Existing database upgrade works
|
||||
- [ ] No duplicate index/table errors
|
||||
- [ ] All tests pass
|
||||
- [ ] Application starts normally
|
||||
- [ ] Can create/read/update notes
|
||||
- [ ] Authentication still works
|
||||
- [ ] Micropub endpoint functional
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: "table already exists" error
|
||||
**Solution**: Check that database_exists_with_tables() is working correctly
|
||||
|
||||
### Issue: "no such column" error
|
||||
**Solution**: Verify INITIAL_SCHEMA_SQL matches v0.1.0 exactly
|
||||
|
||||
### Issue: Migrations not running
|
||||
**Solution**: Check migrations/ directory path and file permissions
|
||||
|
||||
### Issue: Tests failing
|
||||
**Solution**: Ensure test database is properly isolated from production
|
||||
|
||||
## Rollback Procedure
|
||||
|
||||
If issues occur:
|
||||
|
||||
1. Restore database backup
|
||||
2. Revert code changes
|
||||
3. Document issue in ADR-032
|
||||
4. Re-plan implementation
|
||||
|
||||
## Post-Implementation
|
||||
|
||||
1. Update CHANGELOG.md
|
||||
2. Update version number to 1.1.0-rc.1
|
||||
3. Create release notes
|
||||
4. Test Docker container with new schema
|
||||
5. Document any discovered edge cases
|
||||
|
||||
## Contact for Questions
|
||||
|
||||
If you encounter issues not covered in this guide:
|
||||
|
||||
1. Review ADR-031 and ADR-032
|
||||
2. Check existing migration test cases
|
||||
3. Review git history for database.py evolution
|
||||
4. Document any new findings in /docs/reports/
|
||||
|
||||
---
|
||||
|
||||
*Created: 2025-11-24*
|
||||
*For: StarPunk v1.1.0*
|
||||
*Priority: CRITICAL*
|
||||
Reference in New Issue
Block a user