Files
StarPunk/docs/design/v2.8.1/2026-11-03-migration-lock.md
T
PhilandClaude Opus 5.5 8c51b52ed6 fix: Hold the migration lock for the whole run (v2.8.1)
create_migrations_table() committed right after BEGIN IMMEDIATE, and
executescript() commits before running, so gunicorn workers applied
migrations concurrently; the v2.8.0 deploy failed to boot once with
'duplicate column name'. Run migration statements with conn.execute inside
the transaction and don't commit when creating the tracking table.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 09:58:20 -07:00

1.3 KiB

v2.8.1: Migration lock held for the whole run

Incident

Deploying v2.8.0 (migration 014 adds two columns): one gunicorn worker applied 014, another failed with duplicate column name: source_post_url, gunicorn shut down ("Worker failed to boot") and Docker restarted the container. About two seconds of downtime; schema correct afterwards.

Causes

run_migrations takes BEGIN IMMEDIATE so one worker migrates at a time, but the lock was released early twice:

  1. create_migrations_table(), called right after BEGIN IMMEDIATE, ended with conn.commit(). Everything after it ran without the lock.
  2. conn.executescript() COMMITs any open transaction before running the script, so the lock was dropped before the migration was recorded in schema_migrations.

Fix

  • create_migrations_table(conn, commit=True); run_migrations passes commit=False.
  • execute_in_transaction() runs each statement from split_sql_statements() (built on sqlite3.complete_statement, so trigger bodies stay whole) with conn.execute, inside the transaction.

Verification

Four processes migrating a database at 013 simultaneously: 6/30 trials failed before, 0/30 after. tests/test_migration_concurrency.py runs this five times, plus a rollback test showing a migration commits nothing early.