phil/sneakyklaus
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add Participant and MagicToken models with automatic migrations

Browse Source 
Implements Phase 2 infrastructure for participant registration and authentication:

Database Models:
- Add Participant model with exchange scoping and soft deletes
- Add MagicToken model for passwordless authentication
- Add participants relationship to Exchange model
- Include proper indexes and foreign key constraints

Migration Infrastructure:
- Generate Alembic migration for new models
- Create entrypoint.sh script for automatic migrations on container startup
- Update Containerfile to use entrypoint script and include uv binary
- Remove db.create_all() in favor of migration-based schema management

This establishes the foundation for implementing stories 4.1-4.3, 5.1-5.3, and 10.1.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Phil Skentelbery
2025-12-22 16:23:47 -07:00
parent 5201b2f036
commit eaafa78cf3
22 changed files with 10459 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

369
docs/decisions/0003-participant-session-scoping.md Normal file
View File

@@ -0,0 +1,369 @@
# 0003. Participant Session Scoping
Date: 2025-12-22
## Status
Accepted
## Context
Participants in Sneaky Klaus can register for multiple Secret Santa exchanges using the same email address. Each exchange is independent, and we need to decide how to handle authentication and sessions when a participant belongs to multiple exchanges.
### Requirements
1. **Privacy**: Participants in one exchange should not see data from other exchanges
2. **Simplicity**: Authentication should remain frictionless (magic links)
3. **Independence**: Each exchange operates independently
4. **Security**: Sessions must be properly isolated
### Options Considered
We evaluated three approaches for handling participants across multiple exchanges:
## Decision
We will implement **exchange-scoped sessions** where:
1. Each participant registration creates a separate `Participant` record per exchange
2. Each magic link creates a session scoped to a single exchange
3. Participant data and matches are isolated per exchange
4. To access a different exchange, participant must use that exchange's magic link
### Rationale
**Separate Participant Records**:
- Alice registering for "Family Christmas" and "Office Party" creates two distinct `Participant` records
- Each record has its own ID, name, gift ideas, and preferences
- Email is the same, but records are independent
- Simple data model with clear foreign key relationships
**Exchange-Scoped Sessions**:
- Magic link authentication creates session with: `{'user_id': participant_id, 'exchange_id': exchange_id}`
- Session grants access only to the associated exchange
- Participant cannot view or modify data from other exchanges in same session
- Clean security boundary
**Multiple Exchanges Require Multiple Logins**:
- Participant must authenticate separately for each exchange
- Each exchange's magic link creates a new session (replacing previous session)
- No "switch exchange" functionality - use appropriate magic link
- Simple to implement and reason about
## Consequences
### Positive
- **Security**: Clear isolation between exchanges; no risk of data leakage
- **Simplicity**: Straightforward implementation with no complex multi-exchange logic
- **Data Model**: Clean foreign key relationships; each participant belongs to exactly one exchange
- **Privacy**: Participants in Exchange A cannot discover participants in Exchange B
- **Scalability**: No need for complex access control lists or permission systems
- **Testing**: Easy to test; each exchange operates independently
### Negative
- **User Experience**: Participant in multiple exchanges must keep multiple magic links
- **Email Volume**: Separate confirmation emails for each exchange registration
- **No Unified View**: Participant cannot see all their exchanges in one dashboard
- **Duplicate Data**: Same participant name/preferences stored multiple times
### Neutral
- **Email Address**: Same email can appear in multiple exchanges (expected behavior)
- **Session Management**: Only one active participant session at a time (last magic link wins)
- **Magic Link Storage**: Participant should save/bookmark magic links for each exchange
## Implementation Details
### Database Schema
```python
# Two separate Participant records for Alice in two exchanges
Participant(
id=100,
exchange_id=1, # Family Christmas
email="alice@example.com",
name="Alice Smith",
gift_ideas="Books"
)
Participant(
id=200,
exchange_id=2, # Office Party
email="alice@example.com",
name="Alice Smith",
gift_ideas="Coffee mug"
)
```
### Session Structure
```python
# Family Christmas session
session = {
'user_id': 100,
'user_type': 'participant',
'exchange_id': 1
}
# Office Party session (replaces Family Christmas session)
session = {
'user_id': 200,
'user_type': 'participant',
'exchange_id': 2
}
```
### Route Protection
```python
@app.route('/participant/exchange/<int:exchange_id>')
@participant_required
@exchange_access_required
def view_exchange(exchange_id):
"""
Participant can only view exchange if:
1. They are authenticated (participant_required)
2. Their session's exchange_id matches the route's exchange_id
"""
# g.participant.exchange_id must equal exchange_id
# Otherwise: 403 Forbidden
```
## User Experience Implications
### Registration Email
When Alice registers for both exchanges, she receives two emails:
**Email 1** (Family Christmas):
```
Subject: Welcome to Family Christmas!
Hi Alice,
You've successfully registered for the Secret Santa exchange!
[Access My Registration] (magic link for Family Christmas)
```
**Email 2** (Office Party):
```
Subject: Welcome to Office Party!
Hi Alice,
You've successfully registered for the Secret Santa exchange!
[Access My Registration] (magic link for Office Party)
```
### Accessing Multiple Exchanges
**Scenario**: Alice clicks "Family Christmas" magic link, views her assignment, then clicks "Office Party" magic link.
**Behavior**:
1. "Family Christmas" link creates session with exchange_id=1
2. Alice views Family Christmas dashboard
3. "Office Party" link creates NEW session with exchange_id=2 (replaces previous)
4. Alice now views Office Party dashboard
5. To return to Family Christmas, must click Family Christmas magic link again
**Recommendation**: Advise participants to:
- Bookmark magic links for each exchange
- Keep confirmation emails for future access
- Request new magic links anytime via registration page
## Alternatives Considered
### Alternative 1: Unified Multi-Exchange Sessions
**Approach**: Create a single participant identity across all exchanges.
**How it would work**:
- Single `Participant` record per email (not per exchange)
- Many-to-many relationship: Participant ←→ Exchange
- Session grants access to all exchanges for that email
- Dashboard shows all exchanges participant is in
**Why rejected**:
- **Complexity**: Requires many-to-many schema, complex access control
- **Privacy concerns**: Easier to accidentally leak cross-exchange data
- **Name conflicts**: Participant might use different names in different exchanges
- **Preferences diverge**: Gift ideas, reminder settings differ per exchange
- **Admin complexity**: Harder to reason about "removing participant from exchange"
### Alternative 2: Multi-Exchange Sessions with Switching
**Approach**: Session grants access to all exchanges, with UI to switch active exchange.
**How it would work**:
- Separate Participant records (like chosen approach)
- Session contains list of participant_ids: `{'participant_ids': [100, 200]}`
- UI dropdown to "switch active exchange"
- Active exchange stored in session: `{'active_exchange_id': 1}`
**Why rejected**:
- **Complexity**: Session management more complex
- **Security risk**: Easier to introduce bugs that show wrong exchange data
- **Marginal UX benefit**: Switching requires UI action anyway; magic link is simpler
- **Testing burden**: Must test exchange switching logic
- **Session size**: Session grows with number of exchanges
### Alternative 3: No Multiple Exchange Support
**Approach**: Enforce email uniqueness globally across all exchanges.
**How it would work**:
- Email can only be used in one exchange per installation
- Attempting to register with same email in second exchange fails
**Why rejected**:
- **User frustration**: Reasonable to participate in multiple exchanges
- **Workaround temptation**: Users would use alice+family@example.com, alice+work@example.com
- **Use case mismatch**: Common scenario is family member organizing multiple exchanges
## Security Considerations
### Attack Vector: Exchange Data Leakage
**Threat**: Participant in Exchange A attempts to view data from Exchange B.
**Mitigation**:
- All participant routes check `session['exchange_id']` matches route parameter
- Database queries filter by both `participant_id` AND `exchange_id`
- No API or UI allows listing exchanges for an email
**Example Protection**:
```python
@exchange_access_required
def view_exchange(exchange_id):
# Decorator checks: g.participant.exchange_id == exchange_id
# If mismatch: 403 Forbidden, redirect to correct exchange
```
### Attack Vector: Session Hijacking
**Threat**: Attacker steals participant session cookie, accesses exchange.
**Mitigation**:
- Standard session security (HttpOnly, Secure, SameSite=Lax)
- Session scoped to single exchange limits damage
- 7-day session expiration
- No sensitive financial or personal data stored
### Attack Vector: Email Enumeration Across Exchanges
**Threat**: Attacker checks if email is registered in multiple exchanges.
**Mitigation**:
- Magic link request returns generic success message
- No API reveals which exchanges an email is registered in
- Rate limiting prevents automated enumeration
## Testing Strategy
### Unit Tests
```python
def test_participant_isolated_by_exchange():
"""Test that participants are isolated per exchange."""
# Create two exchanges
exchange1 = create_exchange(name="Exchange 1")
exchange2 = create_exchange(name="Exchange 2")
# Register Alice in both
alice1 = register_participant(exchange1.slug, {
'email': 'alice@example.com',
'name': 'Alice',
'gift_ideas': 'Books'
})
alice2 = register_participant(exchange2.slug, {
'email': 'alice@example.com',
'name': 'Alice',
'gift_ideas': 'Coffee'
})
# Different participant IDs
assert alice1.id != alice2.id
# Different exchange IDs
assert alice1.exchange_id == exchange1.id
assert alice2.exchange_id == exchange2.id
def test_session_scoping():
"""Test that session grants access only to associated exchange."""
# Create session for exchange 1
create_participant_session(participant_id=100, exchange_id=1)
# Can access exchange 1
with app.test_client() as client:
response = client.get('/participant/exchange/1')
assert response.status_code == 200
# Cannot access exchange 2
response = client.get('/participant/exchange/2')
assert response.status_code == 403
```
### Integration Tests
```python
def test_multiple_exchange_magic_links():
"""Test magic links for different exchanges create appropriate sessions."""
# Register for exchange 1
token1 = register_and_get_magic_link('exchange1-slug', 'alice@example.com')
# Register for exchange 2
token2 = register_and_get_magic_link('exchange2-slug', 'alice@example.com')
with app.test_client() as client:
# Use token 1
client.get(f'/auth/participant/magic/{token1}')
# Should have access to exchange 1
response = client.get('/participant/exchange/1')
assert response.status_code == 200
# Use token 2 (creates new session)
client.get(f'/auth/participant/magic/{token2}')
# Should now have access to exchange 2
response = client.get('/participant/exchange/2')
assert response.status_code == 200
# No longer have access to exchange 1 (session replaced)
response = client.get('/participant/exchange/1')
assert response.status_code == 403
```
## Future Considerations
### Potential Enhancement: Multi-Exchange Dashboard
If user feedback indicates strong need for unified view, could implement:
**Approach**:
- Add route: `/participant/all` (no auth required, email verification only)
- Participant enters email, receives magic link
- Magic link validates email, shows read-only list of all exchanges for that email
- Each exchange has "Access" button → sends exchange-specific magic link
- Dashboard itself is stateless (no session), just email verification
**Benefits**:
- Participants can see all their exchanges in one place
- Still maintains exchange-scoped sessions for actual data access
- Optional feature; magic links still work independently
**Implementation Complexity**: Medium (new routes, new email template, new UI)
**Recommendation**: Defer until Phase 8 or later based on user feedback
## References
- [ADR-0002: Authentication Strategy](./0002-authentication-strategy.md)
- [Participant Auth Component Design](../designs/v0.2.0/components/participant-auth.md)
- [Data Model v0.2.0](../designs/v0.2.0/data-model.md)
- [Flask Session Documentation](https://flask.palletsprojects.com/en/latest/quickstart/#sessions)

353
docs/decisions/0004-dev-mode-email-logging.md Normal file
View File

@@ -0,0 +1,353 @@
# 0004. Development Mode Email Logging
Date: 2025-12-22
## Status
Accepted
## Context
When testing Sneaky Klaus in development and QA environments without email access, participants cannot receive magic links via email. This blocks QA teams from testing the authentication flow and participant workflows.
### Problem Statement
1. **No Email Service in Dev/QA**: Development environments often don't have access to Resend or external email services
2. **Testing Blocker**: Without magic links, QA cannot test participant registration, authentication, and session flows
3. **Local Development**: Developers need a way to obtain magic links when working locally without email infrastructure
4. **Security Requirement**: This feature MUST be completely disabled in production
### Requirements
1. **Development Only**: Feature must only activate when `FLASK_ENV=development`
2. **Logging**: Full magic link URLs logged to application logs for retrieval
3. **Accessibility**: QA can retrieve links from container logs using standard tools
4. **No Production Exposure**: Absolutely cannot be enabled in production
5. **Clear Documentation**: Must warn developers about security implications
## Decision
We will implement **development mode email logging** where:
1. When `FLASK_ENV=development`, magic link generation logs the complete URL to application logs
2. The full magic link URL (including token) is logged at INFO level with a clear DEV MODE prefix
3. QA retrieves links from container logs using `podman logs` or equivalent tools
4. A runtime check prevents this feature from ever being enabled in production, even if the code exists
5. Email is still sent (when available), but we also provide the logging fallback
### Rationale
**Why Log Instead of Display in UI**:
- Magic links contain 32 bytes of cryptographic randomness - not easily displayed in a UX-friendly way
- Logging to application logs is a standard development practice
- Container logs are easily accessible in QA environments
- No code changes needed for QA to access links
**Why Log to Application Logs**:
- Standard location for operational information
- Automatically captured by container logging systems
- Can be retrieved with `podman logs` without code changes
- Works across all deployment scenarios (local, Docker, Podman, etc.)
**Why Require FLASK_ENV=development**:
- Environment-based gates are best practice for dev-only features
- Clear, explicit control
- Cannot be accidentally enabled in production
- Aligns with Flask conventions
**Why Still Send Email**:
- When email IS available, participants get the real experience
- Logging is a fallback for testing without email
- Both paths work; QA can test with or without actual email service
## Consequences
### Positive
- **Unblocks QA**: Testing can proceed without email infrastructure
- **Developer Friendly**: Developers can work locally without email setup
- **Simple Implementation**: Minimal code changes required
- **Standard Practice**: Logging is a standard development pattern
- **Production Safe**: Environment-based control prevents any production exposure
- **No Breaking Changes**: Existing email workflow unchanged
### Negative
- **Requires Awareness**: QA/developers must know to check logs for links
- **Log Noise**: Application logs include URLs (only in development)
- **Copy-Paste**: Manual link copying from logs is less convenient than email
### Neutral
- **No UI Changes**: No changes to participant-facing experience
- **Email Dependency**: If email IS working, participants still get email
- **Additional State**: No database or session changes needed
## Implementation Details
### Magic Link Logging
When a magic link is generated in development mode, the full URL is logged:
```
[INFO] DEV MODE: Magic link generated for participant user@example.com
[INFO] DEV MODE: Full magic link URL: https://sneaky-klaus.local/auth/participant/magic/k7Jx9mP2qR4sT6vN8bC1dE3fG5hI0jK7lM9nO2pQ4rS6tU8vW1xY3zA5
```
### Environment Check
The logging is guarded by environment check:
```python
import os
def should_log_dev_mode_links() -> bool:
"""
Check if development mode email logging is enabled.
CRITICAL: This must NEVER be True in production.
Only enable when explicitly running with FLASK_ENV=development.
"""
env = os.environ.get('FLASK_ENV', '').lower()
return env == 'development'
```
### Where Links Are Logged
Magic link URLs are logged during generation in the notification service:
```python
def send_registration_confirmation(participant_id: int, token: str):
"""Send registration confirmation email with magic link."""
participant = Participant.query.get(participant_id)
exchange = Exchange.query.get(participant.exchange_id)
# Build magic link URL
magic_link_url = url_for(
'auth.participant_magic_link',
token=token,
_external=True
)
# Development mode: log the link for QA access
if should_log_dev_mode_links():
logger.info(f"DEV MODE: Magic link generated for participant {participant.email}")
logger.info(f"DEV MODE: Full magic link URL: {magic_link_url}")
# Send email (if email service available)
try:
send_email(
to=participant.email,
template='registration_confirmation',
variables={
'participant_name': participant.name,
'exchange_name': exchange.name,
'magic_link_url': magic_link_url,
'exchange_date': exchange.date,
'budget': exchange.budget
}
)
except EmailServiceUnavailable:
# In development, logging fallback ensures we can still test
logger.info(f"Email service unavailable, but link logged for DEV MODE testing")
```
## Security Considerations
### CRITICAL: Production Safety
**This feature MUST NEVER be enabled in production.**
1. **Environment Check**: Code checks `FLASK_ENV` at runtime
2. **No Configuration Flag**: Not configurable; only environment-based
3. **Clear Warning**: Code comments explicitly state the security implication
4. **Immutable in Production**: Production deployments set `FLASK_ENV=production` or unset
5. **Audit Trail**: If logging were somehow enabled in production, it would be obvious in logs
### Attack Vector: Link Interception
**Threat**: Magic links logged in development contain full tokens. If logs are exposed, attacker can use tokens.
**Mitigations**:
- Development environment is not production
- Container logs are only accessible in development/QA networks
- Tokens have 1-hour expiration
- Single-use enforcement prevents replay
- In production, this feature is completely disabled
### Attack Vector: Log Exposure
**Threat**: Production logs somehow exposed and contain dev-mode links.
**Mitigations**:
- Feature only enabled when `FLASK_ENV=development`
- Production environments never set this variable
- If accidentally enabled, logs clearly show "DEV MODE" prefix
- Log aggregation systems should filter dev logs from production
## Testing Strategy
### Unit Tests
```python
def test_dev_mode_logging_disabled_in_production():
"""Ensure development mode logging is never enabled in production."""
# Simulate production environment
os.environ['FLASK_ENV'] = 'production'
# Development mode feature should be disabled
assert should_log_dev_mode_links() is False
def test_dev_mode_logging_enabled_in_development():
"""Ensure development mode logging is enabled in development."""
# Simulate development environment
os.environ['FLASK_ENV'] = 'development'
# Development mode feature should be enabled
assert should_log_dev_mode_links() is True
def test_magic_link_logged_in_dev_mode(caplog):
"""Test that magic links are logged in development mode."""
os.environ['FLASK_ENV'] = 'development'
participant = create_test_participant()
token = generate_magic_token(participant.id, participant.exchange_id)
# Send confirmation (would log the link)
send_registration_confirmation(participant.id, token)
# Check log contains DEV MODE indicator and URL
assert 'DEV MODE' in caplog.text
assert 'magic_link_url' in caplog.text or 'Full magic link URL' in caplog.text
def test_magic_link_not_logged_in_production(caplog):
"""Test that magic links are NOT logged in production."""
os.environ['FLASK_ENV'] = 'production'
participant = create_test_participant()
token = generate_magic_token(participant.id, participant.exchange_id)
# Send confirmation (should NOT log the link)
send_registration_confirmation(participant.id, token)
# Check log does NOT contain DEV MODE indicator
assert 'DEV MODE' not in caplog.text
```
### Integration Tests
```python
def test_qa_workflow_with_dev_mode_logging():
"""
Simulate QA workflow:
1. Register participant
2. Retrieve magic link from logs
3. Use link to authenticate
"""
os.environ['FLASK_ENV'] = 'development'
with app.test_client() as client:
# Step 1: Register participant
response = client.post('/exchange/test-slug/register', data={
'name': 'QA Tester',
'email': 'qa@example.com',
'gift_ideas': 'Testing gifts',
'csrf_token': get_csrf_token()
})
assert response.status_code == 302
# Step 2: Extract magic link from logs
# In real test, would use caplog fixture or retrieve from log file
# For this example, extract from app logs
logs = get_application_logs()
magic_link_match = re.search(r'DEV MODE: Full magic link URL: ([^ ]+)', logs)
assert magic_link_match is not None
magic_link_url = magic_link_match.group(1)
# Extract token from URL
token = magic_link_url.split('/magic/')[-1]
# Step 3: Use magic link to authenticate
response = client.get(f'/auth/participant/magic/{token}')
# Should redirect to dashboard
assert response.status_code == 302
assert 'dashboard' in response.location
# Session should be created
with client.session_transaction() as sess:
assert sess['user_type'] == 'participant'
assert sess['user_id'] is not None
```
## How QA Uses Dev Mode
### For Podman Containers
```bash
# Register a participant via web UI
# Then retrieve the magic link from container logs:
podman logs sneaky-klaus-qa | grep "DEV MODE"
# Output:
# [INFO] DEV MODE: Magic link generated for participant alice@example.com
# [INFO] DEV MODE: Full magic link URL: https://localhost:5000/auth/participant/magic/k7Jx9mP2qR4sT6vN8bC1dE3fG5hI0jK7lM9nO2pQ4rS6tU8vW1xY3zA5
# Copy the URL and paste into browser or use curl:
curl https://localhost:5000/auth/participant/magic/k7Jx9mP2qR4sT6vN8bC1dE3fG5hI0jK7lM9nO2pQ4rS6tU8vW1xY3zA5
```
### For Local Development
```bash
# Run Flask development server with FLASK_ENV=development
FLASK_ENV=development uv run flask run
# Register a participant via web UI
# Magic link appears in terminal output:
# [INFO] DEV MODE: Magic link generated for participant alice@example.com
# [INFO] DEV MODE: Full magic link URL: http://localhost:5000/auth/participant/magic/...
# Copy and use the link
```
## Configuration
No configuration needed. Feature is automatically enabled when:
```
FLASK_ENV=development
```
Feature is automatically disabled when:
```
FLASK_ENV=production
# or unset (defaults to production behavior)
```
## Future Enhancements
### Enhanced Dev Tools
If QA requests additional features:
1. **Dev-Only Endpoint**: Create `/dev/magic-links` endpoint that lists recent links (only in development)
2. **Email Override**: Allow configuration to always log links even if email sends successfully
3. **Log Export**: Endpoint to export recent logs as JSON for automated testing
### Production Monitoring
Consider monitoring/alerting if dev mode logging somehow gets enabled in production:
```python
if should_log_dev_mode_links() and in_production():
send_alert("CRITICAL: Dev mode logging enabled in production!")
```
## References
- [ADR-0002: Authentication Strategy](./0002-authentication-strategy.md)
- [Participant Auth Component Design](../designs/v0.2.0/components/participant-auth.md)
- [Flask Environment and Configuration](https://flask.palletsprojects.com/en/latest/config/)

317
docs/decisions/0005-database-migrations.md Normal file
View File

@@ -0,0 +1,317 @@
# 0005. Database Migrations with Alembic
Date: 2025-12-22
## Status
Accepted
## Context
Sneaky Klaus uses SQLite as its database (see ADR-0001). As the application evolves, the database schema needs to change to support new features. There are two primary approaches to managing database schema:
1. **db.create_all()**: SQLAlchemy's create_all() method creates tables based on current model definitions. Simple but has critical limitations:
- Cannot modify existing tables (add/remove columns, change types)
- Cannot migrate data during schema changes
- No version tracking or rollback capability
- Unsafe for databases with existing data
2. **Schema Migrations**: Tools like Alembic track schema changes as versioned migration files:
- Supports incremental schema changes (add columns, modify constraints, etc.)
- Enables data migrations during schema evolution
- Provides version tracking and rollback capability
- Safe for production databases with existing data
Key considerations:
- Phase 1 (v0.1.0) established Admin and Exchange models
- Phase 2 (v0.2.0) adds Participant and MagicToken models
- Future phases will continue evolving the schema
- Self-hosted deployments may have persistent data from day one
- Users may skip versions or upgrade incrementally
The question is: when should we start using proper database migrations?
## Decision
We will use **Alembic for all database schema changes starting from Phase 2 (v0.2.0)** onward.
Specifically:
1. **Alembic is already configured** in the codebase (alembic.ini, migrations/ directory)
2. **An initial migration already exists** for Admin and Exchange models (created in Phase 1)
3. **All new models and schema changes** will be managed through Alembic migrations
4. **db.create_all() must not be used** for schema creation in production environments
### Migration Workflow
For all schema changes:
1. Modify SQLAlchemy models in `src/models/`
2. Generate migration: `uv run alembic revision --autogenerate -m "description"`
3. Review the generated migration file in `migrations/versions/`
4. Test the migration (upgrade and downgrade paths)
5. Commit the migration file with model changes
6. Apply in deployments: `uv run alembic upgrade head`
### Naming Conventions
Migration messages should be:
- Descriptive and imperative: "Add Participant model", "Add email index to Participant"
- Under 80 characters
- Use lowercase except for model/table names
- Examples:
- "Add Participant and MagicToken models"
- "Add withdrawn_at column to Participant"
- "Create composite index on exchange_id and email"
### Testing Requirements
Every migration must be tested for:
1. **Upgrade path**: `alembic upgrade head` succeeds
2. **Downgrade path**: `alembic downgrade -1` and `alembic upgrade head` both succeed
3. **Schema correctness**: Database schema matches SQLAlchemy model definitions
4. **Application compatibility**: All tests pass after migration
### Handling Existing Databases
For databases created with db.create_all() before migrations were established:
**Option 1 - Stamp (preserves data)**:
```bash
uv run alembic stamp head
```
This marks the database as being at the current migration version without running migrations.
**Option 2 - Recreate (development only)**:
```bash
rm data/sneaky-klaus.db
uv run alembic upgrade head
```
This creates a fresh database from migrations. Only suitable for development.
### Removing db.create_all()
The `db.create_all()` call currently in `src/app.py` should be:
- Removed from production code paths
- Only used in test fixtures where appropriate
- Never used for schema initialization in deployments
Production deployments must use `alembic upgrade head` for schema initialization and updates.
### Automatic Migrations for Self-Hosted Deployments
For self-hosted deployments using containers, migrations must be applied automatically when the container starts. This ensures that:
- Users pulling new container images automatically get schema updates
- No manual migration commands required
- Schema is always in sync with application code
- First-run deployments get proper schema initialization
**Implementation Approach: Container Entrypoint Script**
An entrypoint script runs `alembic upgrade head` before starting the application server. This approach is chosen because:
- **Timing**: Migrations run before application starts, avoiding race conditions
- **Separation of concerns**: Database initialization is separate from application startup
- **Clear error handling**: Migration failures prevent application startup
- **Standard pattern**: Common practice for containerized applications with databases
- **Works with gunicorn**: Gunicorn workers don't need to coordinate migrations
**Entrypoint Script Responsibilities**:
1. Run `alembic upgrade head` to apply all pending migrations
2. Log migration status (success or failure)
3. Exit with error if migrations fail (preventing container startup)
4. Start application server (gunicorn) if migrations succeed
**Implementation**:
```bash
#!/bin/bash
set -e # Exit on any error
echo "Running database migrations..."
if uv run alembic upgrade head; then
echo "Database migrations completed successfully"
else
echo "ERROR: Database migration failed!"
echo "Please check the logs above for details."
exit 1
fi
echo "Starting application server..."
exec gunicorn --bind 0.0.0.0:8000 --workers 2 --threads 4 main:app
```
**Error Handling**:
- Migration failures are logged to stderr
- Container exits with code 1 on migration failure
- Container orchestrator (podman/docker compose) will show failed state
- Users can inspect logs with `podman logs sneaky-klaus` or `docker logs sneaky-klaus`
**Containerfile Changes**:
- Copy entrypoint script: `COPY entrypoint.sh /app/entrypoint.sh`
- Make executable: `RUN chmod +x /app/entrypoint.sh`
- Change CMD to use entrypoint: `CMD ["/app/entrypoint.sh"]`
**First-Run Initialization**:
When no database exists, `alembic upgrade head` will:
1. Create the database file (SQLite)
2. Create the `alembic_version` table to track migration state
3. Run all migrations from scratch
4. Leave database in up-to-date state
**Update Scenarios**:
When updating to a new container image with schema changes:
1. Container starts and runs entrypoint script
2. Alembic detects current schema version from `alembic_version` table
3. Applies only new migrations (incremental upgrade)
4. Application starts with updated schema
**Development Workflow**:
For local development (non-containerized), developers continue to run migrations manually:
```bash
uv run alembic upgrade head
```
This gives developers explicit control over when migrations run during development.
**Alternative Considered: Application Startup Migrations**
Running migrations in `src/app.py` during Flask application startup was considered but rejected:
- **Race conditions**: Multiple gunicorn workers could try to run migrations simultaneously
- **Locking complexity**: Would need migration locks to prevent concurrent runs
- **Startup delays**: Application health checks might fail during migration
- **Error visibility**: Migration failures less visible than container startup failures
- **Not idiomatic**: Flask apps typically don't modify their own schema on startup
The entrypoint script approach is simpler, safer, and more aligned with containerized deployment best practices.
## Consequences
### Positive
- **Safe schema evolution**: Can modify existing tables without data loss
- **Version control**: Schema changes tracked in git alongside code changes
- **Rollback capability**: Can revert problematic schema changes
- **Data migrations**: Can transform data during schema changes (e.g., populate new required columns)
- **Production ready**: Proper migration strategy from the start avoids migration debt
- **Clear deployment process**: `alembic upgrade head` is explicit and auditable
- **Multi-environment support**: Same migrations work across dev, staging, and production
### Negative
- **Additional complexity**: Developers must learn Alembic workflow
- **Migration review required**: Auto-generated migrations must be reviewed for correctness
- **Migration discipline needed**: Schema changes require creating and testing migrations
- **Downgrade path maintenance**: Must write downgrade logic for each migration
- **Linear migration history**: Merge conflicts in migrations can require rebasing
### Neutral
- **Learning curve**: Alembic has good documentation but requires initial learning
- **Migration conflicts**: Multiple developers changing schema simultaneously may need coordination
- **Test database setup**: Tests may need to apply migrations rather than using create_all()
## Implementation Notes
### Phase 2 Implementation
For Phase 2 (v0.2.0), the developer should:
1. Create Participant and MagicToken models in `src/models/`
2. Generate migration:
```bash
uv run alembic revision --autogenerate -m "Add Participant and MagicToken models"
```
3. Review the generated migration file:
- Verify all new tables, columns, and indexes are included
- Check foreign key constraints are correct
- Ensure indexes are created for performance-critical queries
4. Test the migration:
```bash
# Test upgrade
uv run alembic upgrade head
# Test downgrade (optional but recommended)
uv run alembic downgrade -1
uv run alembic upgrade head
```
5. Run application tests to verify compatibility
6. Commit migration file with model changes
### Migration File Structure
Migration files are in `migrations/versions/` and follow this structure:
```python
"""Add Participant and MagicToken models
Revision ID: abc123def456
Revises: eeff6e1a89cd
Create Date: 2025-12-22 10:30:00.000000
"""
from alembic import op
import sqlalchemy as sa
# revision identifiers
revision = 'abc123def456'
down_revision = 'eeff6e1a89cd'
branch_labels = None
depends_on = None
def upgrade():
# Schema changes for upgrade
op.create_table('participant',
sa.Column('id', sa.Integer(), nullable=False),
# ... other columns
)
def downgrade():
# Schema changes for rollback
op.drop_table('participant')
```
### Alembic Configuration
Alembic is configured via `alembic.ini`:
- Migration directory: `migrations/`
- SQLAlchemy URL: Configured dynamically from Flask config in `migrations/env.py`
- Auto-generate support: Enabled
### Documentation Updates
The following documentation has been updated to reflect this decision:
- Phase 2 Implementation Decisions (section 9.1)
- Data Model v0.2.0 (Migration Strategy section)
- System Architecture Overview v0.2.0 (Database Layer section)
## Alternatives Considered
### Continue using db.create_all()
**Rejected**: While simpler initially, db.create_all() cannot handle schema evolution. Since:
- Alembic infrastructure already exists in the codebase
- We expect ongoing schema evolution across multiple phases
- Self-hosted deployments may have persistent data
- Production-ready approach prevents migration debt
Starting with Alembic now is the right choice despite the added complexity.
### Manual SQL migrations
**Rejected**: Writing raw SQL migrations is error-prone and doesn't integrate with SQLAlchemy models. Alembic's autogenerate feature significantly reduces migration creation effort while maintaining safety.
### Django-style migrations
**Rejected**: Django's migration system is tightly coupled to Django ORM. Alembic is the standard for SQLAlchemy-based applications and integrates well with Flask.
### Defer migrations until schema is stable
**Rejected**: The schema will evolve continuously as new features are added. Deferring migrations creates migration debt and makes it harder to support existing deployments. Starting with migrations from Phase 2 establishes good patterns early.
## References
- Alembic documentation: https://alembic.sqlalchemy.org/
- SQLAlchemy documentation: https://docs.sqlalchemy.org/
- ADR-0001: Core Technology Stack
- Phase 2 Implementation Decisions (section 9.1)
- Data Model v0.2.0