ansible/debian-13-upgrade.md

# Plan: upgrade cogitator-01 from Debian 12 (bookworm) to 13 (trixie)

Status: done 2026-11-01. The server runs Debian 13.7, kernel 6.12.113.
The notes below are kept for reference; see "How it went" at the end.

Why: Debian 12 has been LTS-only since June 2026, so fewer packages get
security fixes. Debian 13 is the current stable release.

## Where the server is now (checked 2026-10-31)

- Debian 12.15, kernel 6.1.0-54, Python 3.11, Docker 29.9.2.
- Disk: 237 GB free of 503 GB on `/`, 819 MB free on `/boot`. RAM 15 GB.
- `dpkg --audit` is clean; nothing is held back; one package is upgradable.
- Apt sources: Debian main/updates/security, Docker
  (`download_docker_com_linux_debian.list`) and Tailscale
  (`tailscale.list`, set up by hand, not by Ansible). All say `bookworm`.
- iptables already uses the nftables backend, so ufw and `ufw-docker.sh`
  should behave the same after the upgrade.
- `/tmp` is on the root disk, with no fstab entry.
- Leftovers to tidy: old kernel `linux-image-6.1.0-45`, and config-only
  kernel packages 6.1.0-9/23/25/37.

## Step 1: repo changes (land and deploy before the upgrade)

1. **Replace `apt_key` in `roles/docker/tasks/main.yml`.** Trixie's apt no
   longer ships `apt-key`, so this task would fail on the next deploy.
   Download Docker's key to `/etc/apt/keyrings/docker.asc` with
   `get_url`, and write the repo with `signed-by=`. Use the Debian key URL
   (the task currently fetches the Ubuntu one; it's the same key).
2. **Stop hard-coding `bookworm`.** Use `{{ ansible_distribution_release }}`
   in the Docker repo line, so the role follows whatever release is
   installed. `apt_repository` writes a new file, so remove the old
   `download_docker_com_linux_debian.list` in the same change, to avoid
   having the repo listed twice.
3. Optional: manage the Tailscale repo the same way, so all third-party
   repos come from Ansible.
4. Deploy with `--tags` for the docker role's base tasks, then check that
   `apt update` is clean and nothing changed on bookworm.

## Step 2: backups (the day of the upgrade)

1. **Take a netcup snapshot of the whole server** from the netcup control
   panel. This is the rollback plan. Check that the snapshot has finished
   before going on.
2. Trigger the existing S3 backups by hand (Paperless, GoToSocial), as a
   second copy of the most important data.
3. Note the running containers (`docker ps`) and the dockcheck output, to
   compare against afterwards.

## Step 3: the upgrade (allow about an hour of downtime)

Run in `tmux` over Tailscale SSH, so a dropped connection doesn't kill
apt. Keep the netcup VNC console open as a fallback.

1. Bring bookworm fully up to date: `apt update && apt full-upgrade`,
   reboot if the kernel changed, then `apt autoremove --purge`.
2. Stop the containers, so databases aren't written to while their
   libraries and Docker are replaced: `systemctl stop docker docker.socket`.
3. Change `bookworm` to `trixie` in `/etc/apt/sources.list` (including
   `trixie-security` and `trixie-updates`) and in the Tailscale list. The
   Docker list comes from step 1.
4. `apt update`, then the minimal upgrade first:
   `apt upgrade --without-new-pkgs`, then `apt full-upgrade`.
   - Keep our versions of config files Ansible manages (sshd drop-in,
     `20auto-upgrades`). For anything else, look at the diff before
     choosing.
5. Reboot into kernel 6.12.

## Step 4: checks after the reboot

- `cat /etc/debian_version` shows 13.x; `uname -r` shows 6.12.
- SSH key login works over both the public IP and Tailscale; passwords
  are still refused (`sshd -T | grep -i password`).
- `tailscale status` is up; `ufw status` still has the four Docker
  forward rules (Caddy 80/443, Gitea SSH).
- Docker is running and every container came back (`docker ps` against the
  list from step 2). Check the Postgres and MariaDB/Mongo containers' logs.
- Public sites load through Caddy; ports 22, 80, 443 and 222 are the only
  ones open from outside.
- `/tmp`: trixie can move `/tmp` into RAM (tmpfs) and clean `/tmp` and
  `/var/tmp` on a timer. Check `findmnt /tmp` and make sure nothing here
  keeps files there long-term.
- `unattended-upgrades` picks up `trixie-security` (dry run:
  `unattended-upgrade --dry-run -d`).
- Run `bin/deploy --check` against the whole site to confirm Ansible
  still works with the new Python (3.13) and reports no unexpected changes.
- Then `apt autoremove --purge`, and purge the `rc` kernel packages.

## Rollback

If the server doesn't come back or something important is broken and
can't be fixed quickly, restore the netcup snapshot from step 2. Once
things are stable, delete the snapshot (netcup snapshots slow down the
disk over time).

## Open questions

- Does the netcup plan include snapshots, and how big a snapshot can it
  take? (The server uses 246 GB.)
- When is a good maintenance window?

## How it went (2026-11-01)

Checked first against the trixie release notes (upgrading and issues
chapters). Findings that changed the plan:

- **Don't block service restarts** (no `policy-rc.d`). OpenSSH 9.8+ splits
  sshd into a listener and `sshd-session`; the old listener refuses new
  connections until it's restarted, which would have locked us out before
  the reboot. The script restarted `ssh` after the upgrade instead.
- Interface names: not a risk here. The kernel has `net.ifnames=0`, so the
  NIC stays `eth0`; the address is static and cloud-init's network config
  is disabled. Keep `net.ifnames=0` in `/etc/default/grub` (the new
  `grub.ucf-dist` doesn't have it).
- BIOS GRUB on `/dev/vda`: ran `grub-install /dev/vda && update-grub`
  before rebooting.

Steps run: netcup snapshot, S3 backups (Paperless, GoToSocial), bookworm
fully updated, `/etc` and dpkg state saved to `/root/pre-trixie/`, the
upgrade run as a `systemd-run` unit (log `/root/trixie-upgrade.log`), one
reboot. Downtime about 35 minutes.

Results: all 40 containers back; ufw rules, Tailscale, key-only SSH intact;
public ports still 22, 80, 443, 222; all sites answer; unattended-upgrades
follows trixie; full-site `bin/deploy --check` has no failures.

Afterwards:
- Purged `software-properties-common` (not in trixie; it had pulled in
  PackageKit) and 59 old bookworm libraries.
- `/tmp` is now tmpfs. The upgrade added `/etc/tmpfiles.d/tmp.conf`, which
  keeps the old no-cleanup behaviour.
- systemd's SSH-over-VSOCK generator logged a harmless error on the
  console; it's now turned off by the common role.

Still to do:
- Remove the old 6.1.0-54 kernel once Debian 13 has run fine for a while
  (`apt purge linux-image-6.1.0-54-amd64`).
- Delete the netcup snapshot once things are stable.
- Optional: `apt modernize-sources` to move `sources.list` to deb822.