ansible/debian-13-upgrade.md

Plan: upgrade cogitator-01 from Debian 12 (bookworm) to 13 (trixie)

Status: done 2026-11-01. The server runs Debian 13.7, kernel 6.12.113. The notes below are kept for reference; see "How it went" at the end.

Why: Debian 12 has been LTS-only since June 2026, so fewer packages get security fixes. Debian 13 is the current stable release.

Where the server is now (checked 2026-10-31)

• Debian 12.15, kernel 6.1.0-54, Python 3.11, Docker 29.9.2.
• Disk: 237 GB free of 503 GB on /, 819 MB free on /boot. RAM 15 GB.
• dpkg --audit is clean; nothing is held back; one package is upgradable.
• Apt sources: Debian main/updates/security, Docker (download_docker_com_linux_debian.list) and Tailscale (tailscale.list, set up by hand, not by Ansible). All say bookworm.
• iptables already uses the nftables backend, so ufw and ufw-docker.sh should behave the same after the upgrade.
• /tmp is on the root disk, with no fstab entry.
• Leftovers to tidy: old kernel linux-image-6.1.0-45, and config-only kernel packages 6.1.0-9/23/25/37.

Step 1: repo changes (land and deploy before the upgrade)

1. Replace apt_key in roles/docker/tasks/main.yml. Trixie's apt no longer ships apt-key, so this task would fail on the next deploy. Download Docker's key to /etc/apt/keyrings/docker.asc with get_url, and write the repo with signed-by=. Use the Debian key URL (the task currently fetches the Ubuntu one; it's the same key).
2. Stop hard-coding bookworm. Use {{ ansible_distribution_release }} in the Docker repo line, so the role follows whatever release is installed. apt_repository writes a new file, so remove the old download_docker_com_linux_debian.list in the same change, to avoid having the repo listed twice.
3. Optional: manage the Tailscale repo the same way, so all third-party repos come from Ansible.
4. Deploy with --tags for the docker role's base tasks, then check that apt update is clean and nothing changed on bookworm.

Step 2: backups (the day of the upgrade)

1. Take a netcup snapshot of the whole server from the netcup control panel. This is the rollback plan. Check that the snapshot has finished before going on.
2. Trigger the existing S3 backups by hand (Paperless, GoToSocial), as a second copy of the most important data.
3. Note the running containers (docker ps) and the dockcheck output, to compare against afterwards.

Step 3: the upgrade (allow about an hour of downtime)

Run in tmux over Tailscale SSH, so a dropped connection doesn't kill apt. Keep the netcup VNC console open as a fallback.

1. Bring bookworm fully up to date: apt update && apt full-upgrade, reboot if the kernel changed, then apt autoremove --purge.
2. Stop the containers, so databases aren't written to while their libraries and Docker are replaced: systemctl stop docker docker.socket.
3. Change bookworm to trixie in /etc/apt/sources.list (including trixie-security and trixie-updates) and in the Tailscale list. The Docker list comes from step 1.
4. apt update, then the minimal upgrade first: apt upgrade --without-new-pkgs, then apt full-upgrade.
   • Keep our versions of config files Ansible manages (sshd drop-in, 20auto-upgrades). For anything else, look at the diff before choosing.
5. Reboot into kernel 6.12.

Step 4: checks after the reboot

• cat /etc/debian_version shows 13.x; uname -r shows 6.12.
• SSH key login works over both the public IP and Tailscale; passwords are still refused (sshd -T | grep -i password).
• tailscale status is up; ufw status still has the four Docker forward rules (Caddy 80/443, Gitea SSH).
• Docker is running and every container came back (docker ps against the list from step 2). Check the Postgres and MariaDB/Mongo containers' logs.
• Public sites load through Caddy; ports 22, 80, 443 and 222 are the only ones open from outside.
• /tmp: trixie can move /tmp into RAM (tmpfs) and clean /tmp and /var/tmp on a timer. Check findmnt /tmp and make sure nothing here keeps files there long-term.
• unattended-upgrades picks up trixie-security (dry run: unattended-upgrade --dry-run -d).
• Run bin/deploy --check against the whole site to confirm Ansible still works with the new Python (3.13) and reports no unexpected changes.
• Then apt autoremove --purge, and purge the rc kernel packages.

Rollback

If the server doesn't come back or something important is broken and can't be fixed quickly, restore the netcup snapshot from step 2. Once things are stable, delete the snapshot (netcup snapshots slow down the disk over time).

Open questions

• Does the netcup plan include snapshots, and how big a snapshot can it take? (The server uses 246 GB.)
• When is a good maintenance window?

How it went (2026-11-01)

Checked first against the trixie release notes (upgrading and issues chapters). Findings that changed the plan:

• Don't block service restarts (no policy-rc.d). OpenSSH 9.8+ splits sshd into a listener and sshd-session; the old listener refuses new connections until it's restarted, which would have locked us out before the reboot. The script restarted ssh after the upgrade instead.
• Interface names: not a risk here. The kernel has net.ifnames=0, so the NIC stays eth0; the address is static and cloud-init's network config is disabled. Keep net.ifnames=0 in /etc/default/grub (the new grub.ucf-dist doesn't have it).
• BIOS GRUB on /dev/vda: ran grub-install /dev/vda && update-grub before rebooting.

Steps run: netcup snapshot, S3 backups (Paperless, GoToSocial), bookworm fully updated, /etc and dpkg state saved to /root/pre-trixie/, the upgrade run as a systemd-run unit (log /root/trixie-upgrade.log), one reboot. Downtime about 35 minutes.

Results: all 40 containers back; ufw rules, Tailscale, key-only SSH intact; public ports still 22, 80, 443, 222; all sites answer; unattended-upgrades follows trixie; full-site bin/deploy --check has no failures.

Afterwards:

• Purged software-properties-common (not in trixie; it had pulled in PackageKit) and 59 old bookworm libraries.
• /tmp is now tmpfs. The upgrade added /etc/tmpfiles.d/tmp.conf, which keeps the old no-cleanup behaviour.
• systemd's SSH-over-VSOCK generator logged a harmless error on the console; it's now turned off by the common role.

Still to do:

• Remove the old 6.1.0-54 kernel once Debian 13 has run fine for a while (apt purge linux-image-6.1.0-54-amd64).
• Delete the netcup snapshot once things are stable.
• Optional: apt modernize-sources to move sources.list to deb822.