StarPunk/docs/design/initial-schema-implementation-guide.md

# Initial Schema SQL Implementation Guide

## Overview

This guide provides step-by-step instructions for implementing the INITIAL_SCHEMA_SQL constant and updating the database initialization system as specified in ADR-032.

**Priority**: CRITICAL for v1.1.0
**Estimated Time**: 4-6 hours
**Risk Level**: Low (backward compatible)

## Pre-Implementation Checklist

- [ ] Read ADR-031 (Database Migration System Redesign)
- [ ] Read ADR-032 (Initial Schema SQL Implementation)
- [ ] Review current migrations in `/migrations/` directory
- [ ] Backup any test databases
- [ ] Ensure test environment is ready

## Implementation Steps

### Step 1: Add INITIAL_SCHEMA_SQL Constant

**File**: `/home/phil/Projects/starpunk/starpunk/database.py`

**Action**: Add the following constant ABOVE the current SCHEMA_SQL:

```python
# Database schema - V0.1.0 baseline (see ADR-032)
# This represents the initial database structure from commit a68fd57
# All schema evolution happens through migrations from this baseline
INITIAL_SCHEMA_SQL = """
-- Notes metadata (content is in files)
CREATE TABLE IF NOT EXISTS notes (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    slug TEXT UNIQUE NOT NULL,
    file_path TEXT UNIQUE NOT NULL,
    published BOOLEAN DEFAULT 0,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    deleted_at TIMESTAMP,
    content_hash TEXT
);

CREATE INDEX IF NOT EXISTS idx_notes_created_at ON notes(created_at DESC);
CREATE INDEX IF NOT EXISTS idx_notes_published ON notes(published);
CREATE INDEX IF NOT EXISTS idx_notes_slug ON notes(slug);
CREATE INDEX IF NOT EXISTS idx_notes_deleted_at ON notes(deleted_at);

-- Authentication sessions (IndieLogin)
CREATE TABLE IF NOT EXISTS sessions (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_token TEXT UNIQUE NOT NULL,
    me TEXT NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP NOT NULL,
    last_used_at TIMESTAMP
);

CREATE INDEX IF NOT EXISTS idx_sessions_token ON sessions(session_token);
CREATE INDEX IF NOT EXISTS idx_sessions_expires ON sessions(expires_at);

-- Micropub access tokens (original insecure version)
CREATE TABLE IF NOT EXISTS tokens (
    token TEXT PRIMARY KEY,
    me TEXT NOT NULL,
    client_id TEXT,
    scope TEXT,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP
);

CREATE INDEX IF NOT EXISTS idx_tokens_me ON tokens(me);

-- CSRF state tokens (for IndieAuth flow)
CREATE TABLE IF NOT EXISTS auth_state (
    state TEXT PRIMARY KEY,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP NOT NULL
);

CREATE INDEX IF NOT EXISTS idx_auth_state_expires ON auth_state(expires_at);
"""
```

### Step 2: Rename Current SCHEMA_SQL

**File**: `/home/phil/Projects/starpunk/starpunk/database.py`

**Action**: Rename the existing SCHEMA_SQL constant and add documentation:

```python
# Current database schema - FOR DOCUMENTATION ONLY
# This shows the current complete schema after all migrations
# NOT used for database initialization - see INITIAL_SCHEMA_SQL
# Updated by migrations 001 and 002
CURRENT_SCHEMA_SQL = """
[existing SCHEMA_SQL content]
"""
```

### Step 3: Add Helper Function

**File**: `/home/phil/Projects/starpunk/starpunk/database.py`

**Action**: Add this function before init_db():

```python
def database_exists_with_tables(db_path):
    """
    Check if database exists and has tables

    Args:
        db_path: Path to SQLite database file

    Returns:
        bool: True if database exists with at least one table
    """
    import os

    # Check if file exists
    if not os.path.exists(db_path):
        return False

    # Check if it has tables
    try:
        conn = sqlite3.connect(db_path)
        cursor = conn.execute(
            "SELECT COUNT(*) FROM sqlite_master WHERE type='table'"
        )
        table_count = cursor.fetchone()[0]
        conn.close()
        return table_count > 0
    except Exception:
        return False
```

### Step 4: Update init_db() Function

**File**: `/home/phil/Projects/starpunk/starpunk/database.py`

**Action**: Replace the init_db() function with:

```python
def init_db(app=None):
    """
    Initialize database schema and run migrations

    For fresh databases:
    1. Creates v0.1.0 baseline schema (INITIAL_SCHEMA_SQL)
    2. Runs all migrations to bring to current version

    For existing databases:
    1. Skips schema creation (tables already exist)
    2. Runs only pending migrations

    Args:
        app: Flask application instance (optional, for config access)
    """
    if app:
        db_path = app.config["DATABASE_PATH"]
        logger = app.logger
    else:
        # Fallback to default path
        db_path = Path("./data/starpunk.db")
        logger = logging.getLogger(__name__)

    # Ensure parent directory exists
    db_path.parent.mkdir(parents=True, exist_ok=True)

    # Check if this is an existing database
    if database_exists_with_tables(db_path):
        # Existing database - skip schema creation, only run migrations
        logger.info(f"Existing database found: {db_path}")
        logger.info("Running pending migrations...")
    else:
        # Fresh database - create initial v0.1.0 schema
        logger.info(f"Creating new database: {db_path}")
        conn = sqlite3.connect(db_path)
        try:
            # Create v0.1.0 baseline schema
            conn.executescript(INITIAL_SCHEMA_SQL)
            conn.commit()
            logger.info("Created initial v0.1.0 database schema")
        except Exception as e:
            logger.error(f"Failed to create initial schema: {e}")
            raise
        finally:
            conn.close()

    # Run migrations (for both fresh and existing databases)
    # This will apply ALL migrations for fresh databases,
    # or only pending migrations for existing databases
    from starpunk.migrations import run_migrations

    try:
        run_migrations(db_path, logger)
    except Exception as e:
        logger.error(f"Migration failed: {e}")
        raise
```

### Step 5: Update Tests

**File**: `/home/phil/Projects/starpunk/tests/test_migrations.py`

**Add these test cases**:

```python
def test_fresh_database_initialization(tmp_path):
    """Test that fresh database gets initial schema then migrations"""
    db_path = tmp_path / "test.db"

    # Initialize fresh database
    init_db_with_path(db_path)

    # Verify initial tables exist
    conn = sqlite3.connect(db_path)
    cursor = conn.execute(
        "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
    )
    tables = [row[0] for row in cursor.fetchall()]

    # Should have all tables including migration tracking
    assert "notes" in tables
    assert "sessions" in tables
    assert "tokens" in tables
    assert "auth_state" in tables
    assert "schema_migrations" in tables
    assert "authorization_codes" in tables  # Added by migration 002

    # Verify migrations were applied
    cursor = conn.execute("SELECT COUNT(*) FROM schema_migrations")
    migration_count = cursor.fetchone()[0]
    assert migration_count >= 2  # At least migrations 001 and 002

    conn.close()


def test_existing_database_upgrade(tmp_path):
    """Test that existing database only runs pending migrations"""
    db_path = tmp_path / "test.db"

    # Create a database with v0.1.0 schema manually
    conn = sqlite3.connect(db_path)
    conn.executescript(INITIAL_SCHEMA_SQL)
    conn.commit()
    conn.close()

    # Run init_db on existing database
    init_db_with_path(db_path)

    # Verify migrations were applied
    conn = sqlite3.connect(db_path)

    # Check that migration 001 was applied (code_verifier column)
    cursor = conn.execute("PRAGMA table_info(auth_state)")
    columns = [row[1] for row in cursor.fetchall()]
    assert "code_verifier" in columns

    # Check that migration 002 was applied (authorization_codes table)
    cursor = conn.execute(
        "SELECT name FROM sqlite_master WHERE type='table' AND name='authorization_codes'"
    )
    assert cursor.fetchone() is not None

    conn.close()
```

### Step 6: Manual Testing Procedure

1. **Test Fresh Database**:
   ```bash
   # Backup existing database
   mv data/starpunk.db data/starpunk.db.backup

   # Start application (will create fresh database)
   uv run python app.py

   # Verify application starts without errors
   # Check logs for "Created initial v0.1.0 database schema"
   # Check logs for "Applied migration: 001_add_code_verifier_to_auth_state.sql"
   # Check logs for "Applied migration: 002_secure_tokens_and_authorization_codes.sql"
   ```

2. **Test Existing Database**:
   ```bash
   # Restore backup
   cp data/starpunk.db.backup data/starpunk.db

   # Start application
   uv run python app.py

   # Verify application starts without errors
   # Check logs for "Existing database found"
   # Check logs for migration status
   ```

3. **Test Database Queries**:
   ```bash
   sqlite3 data/starpunk.db

   # Check tables
   .tables

   # Check schema_migrations
   SELECT * FROM schema_migrations;

   # Verify authorization_codes table exists
   .schema authorization_codes

   # Verify tokens table has token_hash column
   .schema tokens
   ```

### Step 7: Update Documentation

**File**: `/home/phil/Projects/starpunk/docs/architecture/database.md`

**Add section**:

```markdown
## Schema Evolution Strategy

StarPunk uses a baseline + migrations approach for schema management:

1. **INITIAL_SCHEMA_SQL**: Represents the v0.1.0 baseline schema
2. **Migrations**: All schema changes applied sequentially
3. **CURRENT_SCHEMA_SQL**: Documentation of current complete schema

This ensures:
- Predictable upgrade paths from any version
- Clear schema history through migrations
- Testable database evolution
```

## Validation Checklist

After implementation, verify:

- [ ] Fresh database initialization works
- [ ] Existing database upgrade works
- [ ] No duplicate index/table errors
- [ ] All tests pass
- [ ] Application starts normally
- [ ] Can create/read/update notes
- [ ] Authentication still works
- [ ] Micropub endpoint functional

## Troubleshooting

### Issue: "table already exists" error
**Solution**: Check that database_exists_with_tables() is working correctly

### Issue: "no such column" error
**Solution**: Verify INITIAL_SCHEMA_SQL matches v0.1.0 exactly

### Issue: Migrations not running
**Solution**: Check migrations/ directory path and file permissions

### Issue: Tests failing
**Solution**: Ensure test database is properly isolated from production

## Rollback Procedure

If issues occur:

1. Restore database backup
2. Revert code changes
3. Document issue in ADR-032
4. Re-plan implementation

## Post-Implementation

1. Update CHANGELOG.md
2. Update version number to 1.1.0-rc.1
3. Create release notes
4. Test Docker container with new schema
5. Document any discovered edge cases

## Contact for Questions

If you encounter issues not covered in this guide:

1. Review ADR-031 and ADR-032
2. Check existing migration test cases
3. Review git history for database.py evolution
4. Document any new findings in /docs/reports/

---

*Created: 2025-11-24*
*For: StarPunk v1.1.0*
*Priority: CRITICAL*