sneakyklaus/docs/designs/v0.2.0/automatic-migration-implementation.md

# Automatic Migration Implementation Guide

**Version**: 0.2.0
**Date**: 2025-12-22
**Status**: Implementation Specification

## Overview

This document specifies how to implement automatic database migrations for containerized deployments of Sneaky Klaus. When self-hosted users pull a new container image with schema changes, migrations must be applied automatically without manual intervention.

## Context

- **Current State**: Container runs gunicorn directly; no automatic migrations
- **Problem**: Users must manually run `alembic upgrade head` after pulling new images
- **Solution**: Container entrypoint script applies migrations before starting app
- **Reference**: ADR-0005 "Automatic Migrations for Self-Hosted Deployments"

## Implementation Steps

### Step 1: Create Entrypoint Script

Create `/home/phil/Projects/sneaky-klaus/entrypoint.sh`:

```bash
#!/bin/bash
set -e  # Exit on any error

echo "Running database migrations..."
if uv run alembic upgrade head; then
    echo "Database migrations completed successfully"
else
    echo "ERROR: Database migration failed!"
    echo "Please check the logs above for details."
    exit 1
fi

echo "Starting application server..."
exec gunicorn --bind 0.0.0.0:8000 --workers 2 --threads 4 main:app
```

**Key Points**:
- `set -e`: Ensures script exits immediately if any command fails
- Migrations run before gunicorn starts
- `exec` replaces the shell process with gunicorn (PID 1 becomes gunicorn for proper signal handling)
- Clear logging for debugging
- Exit code 1 on migration failure prevents container from starting

### Step 2: Update Containerfile

Modify `/home/phil/Projects/sneaky-klaus/Containerfile`:

**Add after line 29 (after copying application code)**:
```dockerfile
# Copy entrypoint script
COPY entrypoint.sh ./
RUN chmod +x entrypoint.sh
```

**Replace line 51 (CMD line)**:

From:
```dockerfile
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "2", "--threads", "4", "main:app"]
```

To:
```dockerfile
CMD ["./entrypoint.sh"]
```

**Full context** (lines 27-52 after changes):
```dockerfile
# Copy application code
COPY src/ ./src/
COPY migrations/ ./migrations/
COPY alembic.ini main.py ./

# Copy entrypoint script
COPY entrypoint.sh ./
RUN chmod +x entrypoint.sh

# Set environment variables
ENV PATH="/app/.venv/bin:$PATH"
ENV PYTHONUNBUFFERED=1
ENV PYTHONDONTWRITEBYTECODE=1
ENV FLASK_ENV=production

# Create data directory and set permissions
RUN mkdir -p /app/data && chown -R appuser:appuser /app

# Switch to non-root user
USER appuser

# Expose port
EXPOSE 8000

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD python -c "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')" || exit 1

# Run with entrypoint script
CMD ["./entrypoint.sh"]
```

### Step 3: Remove db.create_all() from Application

Modify `/home/phil/Projects/sneaky-klaus/src/app.py`:

**Replace lines 74-77**:

From:
```python
# Import models to ensure they're registered with SQLAlchemy
with app.app_context():
    from src.models import Admin, Exchange, RateLimit  # noqa: F401

    db.create_all()
```

To:
```python
# Import models to ensure they're registered with SQLAlchemy
with app.app_context():
    from src.models import Admin, Exchange, RateLimit  # noqa: F401
    # Schema managed by Alembic migrations (applied via entrypoint script)
```

**Rationale**:
- Migrations are now handled by entrypoint script
- `db.create_all()` conflicts with migration-based schema management
- `db.create_all()` cannot handle schema changes (only creates missing tables)
- ADR-0005 explicitly requires migrations for production

### Step 4: Copy uv to Runtime Stage

The entrypoint script uses `uv run alembic`, so we need `uv` available in the runtime stage.

Modify `/home/phil/Projects/sneaky-klaus/Containerfile`:

**Add after line 23 (after copying .venv from builder)**:

```dockerfile
# Copy uv for running alembic in entrypoint
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
```

**Full context** (lines 15-30 after change):
```dockerfile
# Runtime stage - minimal image
FROM python:3.12-slim AS runtime

WORKDIR /app

# Create non-root user
RUN useradd --create-home --shell /bin/bash appuser

# Copy virtual environment from builder
COPY --from=builder /app/.venv /app/.venv

# Copy uv for running alembic in entrypoint
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv

# Copy application code
COPY src/ ./src/
COPY migrations/ ./migrations/
COPY alembic.ini main.py ./
```

## Testing

### Test 1: First-Run Scenario

Verify migrations run when starting with no database:

```bash
# Build new image
podman build -t sneaky-klaus:test .

# Remove any existing data volume
podman volume rm sneaky-klaus-data || true

# Start container
podman run -p 8000:8000 \
  -e SECRET_KEY=test-secret-key \
  -e APP_URL=http://localhost:8000 \
  -v sneaky-klaus-data:/app/data \
  --name sneaky-klaus-test \
  sneaky-klaus:test

# Check logs - should see migration messages
podman logs sneaky-klaus-test

# Expected output:
# Running database migrations...
# INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
# INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
# INFO  [alembic.runtime.migration] Running upgrade  -> eeff6e1a89cd, initial schema with Admin and Exchange models
# Database migrations completed successfully
# Starting application server...
```

### Test 2: Update Scenario

Verify incremental migrations run when updating:

```bash
# With container already running from Test 1, rebuild with new migration
# (After developer adds new migration in Phase 2)
podman build -t sneaky-klaus:test .

# Stop and remove old container
podman stop sneaky-klaus-test
podman rm sneaky-klaus-test

# Start new container with same data volume
podman run -p 8000:8000 \
  -e SECRET_KEY=test-secret-key \
  -e APP_URL=http://localhost:8000 \
  -v sneaky-klaus-data:/app/data \
  --name sneaky-klaus-test \
  sneaky-klaus:test

# Check logs - should see new migration applied
podman logs sneaky-klaus-test

# Expected output:
# Running database migrations...
# INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
# INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
# INFO  [alembic.runtime.migration] Running upgrade eeff6e1a89cd -> abc123def456, Add Participant and MagicToken models
# Database migrations completed successfully
# Starting application server...
```

### Test 3: Migration Failure Scenario

Verify container fails to start if migration fails:

```bash
# Manually corrupt the database or create an invalid migration
# Container should exit with error code 1
podman logs sneaky-klaus-test

# Expected output:
# Running database migrations...
# ERROR: Database migration failed!
# Please check the logs above for details.

# Container should not be running
podman ps -a | grep sneaky-klaus-test
# Should show "Exited (1)"
```

### Test 4: Development Workflow

Verify developers can still run migrations manually:

```bash
# Local development (no container)
uv run alembic upgrade head

# Should work as before
```

## Rollout Plan

### Phase 1: Update Current Release (v0.1.0)

Since v0.1.0 is already released, we should add automatic migrations to ensure users have this feature before Phase 2 ships:

1. Create entrypoint script
2. Update Containerfile
3. Remove `db.create_all()` from app.py
4. Test with existing v0.1.0 schema
5. Merge to `release/v0.1.0` branch
6. Build and tag new v0.1.0 patch release (v0.1.1)
7. Document in release notes

### Phase 2: Available for New Migrations (v0.2.0)

When Phase 2 development starts:

1. New Participant/MagicToken migrations will use automatic migration system
2. Test that migrations apply automatically on container startup
3. Document behavior in Phase 2 release notes

## Error Scenarios

### Scenario 1: Migration Fails

**Symptom**: Container exits immediately after startup
**User Action**:
1. Check logs: `podman logs sneaky-klaus`
2. Identify failing migration from Alembic output
3. Report issue or fix database manually
4. Restart container

**Prevention**: Thorough testing of migrations before release

### Scenario 2: Database File Permissions

**Symptom**: Alembic cannot write to database file
**Root Cause**: Volume mount permissions mismatch
**User Action**: Ensure data volume is writable by container user (UID 1000)

### Scenario 3: Missing uv Binary

**Symptom**: `/bin/bash: uv: command not found`
**Root Cause**: uv not copied to runtime stage
**Fix**: Verify Containerfile includes uv COPY step (Step 4 above)

## Documentation Updates

### User Documentation

Update deployment documentation to explain:
- Migrations run automatically on container startup
- How to check migration logs
- What to do if migrations fail
- No manual migration commands needed

### Developer Documentation

Update CLAUDE.md and development guide:
- Container entrypoint runs migrations
- Local development still uses manual `uv run alembic upgrade head`
- How to test migrations in container

## Future Enhancements

Potential improvements for future phases:

### Backup Before Migration

Add automatic backup before running migrations:

```bash
#!/bin/bash
set -e

echo "Backing up database..."
if [ -f /app/data/sneaky-klaus.db ]; then
    cp /app/data/sneaky-klaus.db /app/data/sneaky-klaus.db.backup-$(date +%Y%m%d-%H%M%S)
fi

echo "Running database migrations..."
# ... rest of script
```

### Migration Dry-Run Mode

Add environment variable for dry-run mode:

```bash
if [ "$MIGRATION_DRY_RUN" = "true" ]; then
    echo "DRY RUN MODE: Showing pending migrations..."
    uv run alembic current
    uv run alembic heads
    exit 0
fi
```

### Migration Timeout

Add timeout to prevent hanging:

```bash
timeout 300 uv run alembic upgrade head || {
    echo "ERROR: Migration timed out after 5 minutes"
    exit 1
}
```

These enhancements are not required for Phase 2 but could be considered for Phase 5 (Operations) or later.

## References

- ADR-0005: Database Migrations with Alembic
- Phase 2 Implementation Decisions (Section 9.2)
- Alembic Documentation: https://alembic.sqlalchemy.org/
- Docker/Podman Entrypoint Best Practices

## Acceptance Criteria

This implementation is complete when:

- [ ] `entrypoint.sh` created and executable
- [ ] Containerfile updated to use entrypoint script
- [ ] `uv` binary available in runtime stage
- [ ] `db.create_all()` removed from `src/app.py`
- [ ] First-run test passes (fresh database)
- [ ] Update test passes (existing database with new migration)
- [ ] Failure test passes (container exits on migration error)
- [ ] Documentation updated
- [ ] Changes merged to release branch