From 9b4df20080f779ee100047b7b588531d9918ada1 Mon Sep 17 00:00:00 2001 From: Phil Date: Sat, 31 Oct 2026 17:47:22 -0600 Subject: [PATCH] Draft the Debian 13 upgrade plan Co-Authored-By: Claude Opus 5.5 --- debian-13-upgrade.md | 99 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100644 debian-13-upgrade.md diff --git a/debian-13-upgrade.md b/debian-13-upgrade.md new file mode 100644 index 0000000..5fa88ae --- /dev/null +++ b/debian-13-upgrade.md @@ -0,0 +1,99 @@ +# Plan: upgrade cogitator-01 from Debian 12 (bookworm) to 13 (trixie) + +Status: draft, started 2026-10-31. Nothing here has been run yet. + +Why: Debian 12 has been LTS-only since June 2026, so fewer packages get +security fixes. Debian 13 is the current stable release. + +## Where the server is now (checked 2026-10-31) + +- Debian 12.15, kernel 6.1.0-54, Python 3.11, Docker 29.9.2. +- Disk: 237 GB free of 503 GB on `/`, 819 MB free on `/boot`. RAM 15 GB. +- `dpkg --audit` is clean; nothing is held back; one package is upgradable. +- Apt sources: Debian main/updates/security, Docker + (`download_docker_com_linux_debian.list`) and Tailscale + (`tailscale.list`, set up by hand, not by Ansible). All say `bookworm`. +- iptables already uses the nftables backend, so ufw and `ufw-docker.sh` + should behave the same after the upgrade. +- `/tmp` is on the root disk, with no fstab entry. +- Leftovers to tidy: old kernel `linux-image-6.1.0-45`, and config-only + kernel packages 6.1.0-9/23/25/37. + +## Step 1: repo changes (land and deploy before the upgrade) + +1. **Replace `apt_key` in `roles/docker/tasks/main.yml`.** Trixie's apt no + longer ships `apt-key`, so this task would fail on the next deploy. + Download Docker's key to `/etc/apt/keyrings/docker.asc` with + `get_url`, and write the repo with `signed-by=`. Use the Debian key URL + (the task currently fetches the Ubuntu one; it's the same key). +2. **Stop hard-coding `bookworm`.** Use `{{ ansible_distribution_release }}` + in the Docker repo line, so the role follows whatever release is + installed. `apt_repository` writes a new file, so remove the old + `download_docker_com_linux_debian.list` in the same change, to avoid + having the repo listed twice. +3. Optional: manage the Tailscale repo the same way, so all third-party + repos come from Ansible. +4. Deploy with `--tags` for the docker role's base tasks, then check that + `apt update` is clean and nothing changed on bookworm. + +## Step 2: backups (the day of the upgrade) + +1. **Take a netcup snapshot of the whole server** from the netcup control + panel. This is the rollback plan. Check that the snapshot has finished + before going on. +2. Trigger the existing S3 backups by hand (Paperless, GoToSocial), as a + second copy of the most important data. +3. Note the running containers (`docker ps`) and the dockcheck output, to + compare against afterwards. + +## Step 3: the upgrade (allow about an hour of downtime) + +Run in `tmux` over Tailscale SSH, so a dropped connection doesn't kill +apt. Keep the netcup VNC console open as a fallback. + +1. Bring bookworm fully up to date: `apt update && apt full-upgrade`, + reboot if the kernel changed, then `apt autoremove --purge`. +2. Stop the containers, so databases aren't written to while their + libraries and Docker are replaced: `systemctl stop docker docker.socket`. +3. Change `bookworm` to `trixie` in `/etc/apt/sources.list` (including + `trixie-security` and `trixie-updates`) and in the Tailscale list. The + Docker list comes from step 1. +4. `apt update`, then the minimal upgrade first: + `apt upgrade --without-new-pkgs`, then `apt full-upgrade`. + - Keep our versions of config files Ansible manages (sshd drop-in, + `20auto-upgrades`). For anything else, look at the diff before + choosing. +5. Reboot into kernel 6.12. + +## Step 4: checks after the reboot + +- `cat /etc/debian_version` shows 13.x; `uname -r` shows 6.12. +- SSH key login works over both the public IP and Tailscale; passwords + are still refused (`sshd -T | grep -i password`). +- `tailscale status` is up; `ufw status` still has the four Docker + forward rules (Caddy 80/443, Gitea SSH). +- Docker is running and every container came back (`docker ps` against the + list from step 2). Check the Postgres and MariaDB/Mongo containers' logs. +- Public sites load through Caddy; ports 22, 80, 443 and 222 are the only + ones open from outside. +- `/tmp`: trixie can move `/tmp` into RAM (tmpfs) and clean `/tmp` and + `/var/tmp` on a timer. Check `findmnt /tmp` and make sure nothing here + keeps files there long-term. +- `unattended-upgrades` picks up `trixie-security` (dry run: + `unattended-upgrade --dry-run -d`). +- Run `bin/deploy --check` against the whole site to confirm Ansible + still works with the new Python (3.13) and reports no unexpected changes. +- Then `apt autoremove --purge`, and purge the `rc` kernel packages. + +## Rollback + +If the server doesn't come back or something important is broken and +can't be fixed quickly, restore the netcup snapshot from step 2. Once +things are stable, delete the snapshot (netcup snapshots slow down the +disk over time). + +## Open questions + +- Does the netcup plan include snapshots, and how big a snapshot can it + take? (The server uses 246 GB.) +- When is a good maintenance window?