create_migrations_table() committed right after BEGIN IMMEDIATE, and executescript() commits before running, so gunicorn workers applied migrations concurrently; the v2.8.0 deploy failed to boot once with 'duplicate column name'. Run migration statements with conn.execute inside the transaction and don't commit when creating the tracking table. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1.3 KiB
1.3 KiB
v2.8.1: Migration lock held for the whole run
Incident
Deploying v2.8.0 (migration 014 adds two columns): one gunicorn worker
applied 014, another failed with duplicate column name: source_post_url,
gunicorn shut down ("Worker failed to boot") and Docker restarted the
container. About two seconds of downtime; schema correct afterwards.
Causes
run_migrations takes BEGIN IMMEDIATE so one worker migrates at a time,
but the lock was released early twice:
create_migrations_table(), called right afterBEGIN IMMEDIATE, ended withconn.commit(). Everything after it ran without the lock.conn.executescript()COMMITs any open transaction before running the script, so the lock was dropped before the migration was recorded inschema_migrations.
Fix
create_migrations_table(conn, commit=True);run_migrationspassescommit=False.execute_in_transaction()runs each statement fromsplit_sql_statements()(built onsqlite3.complete_statement, so trigger bodies stay whole) withconn.execute, inside the transaction.
Verification
Four processes migrating a database at 013 simultaneously: 6/30 trials
failed before, 0/30 after. tests/test_migration_concurrency.py runs this
five times, plus a rollback test showing a migration commits nothing early.