# Plan: upgrade cogitator-01 from Debian 12 (bookworm) to 13 (trixie)

Status: draft, started 2026-10-31. Nothing here has been run yet.

Why: Debian 12 has been LTS-only since June 2026, so fewer packages get
security fixes. Debian 13 is the current stable release.

## Where the server is now (checked 2026-10-31)

- Debian 12.15, kernel 6.1.0-54, Python 3.11, Docker 29.9.2.
- Disk: 237 GB free of 503 GB on `/`, 819 MB free on `/boot`. RAM 15 GB.
- `dpkg --audit` is clean; nothing is held back; one package is upgradable.
- Apt sources: Debian main/updates/security, Docker
  (`download_docker_com_linux_debian.list`) and Tailscale
  (`tailscale.list`, set up by hand, not by Ansible). All say `bookworm`.
- iptables already uses the nftables backend, so ufw and `ufw-docker.sh`
  should behave the same after the upgrade.
- `/tmp` is on the root disk, with no fstab entry.
- Leftovers to tidy: old kernel `linux-image-6.1.0-45`, and config-only
  kernel packages 6.1.0-9/23/25/37.

## Step 1: repo changes (land and deploy before the upgrade)

1. **Replace `apt_key` in `roles/docker/tasks/main.yml`.** Trixie's apt no
   longer ships `apt-key`, so this task would fail on the next deploy.
   Download Docker's key to `/etc/apt/keyrings/docker.asc` with
   `get_url`, and write the repo with `signed-by=`. Use the Debian key URL
   (the task currently fetches the Ubuntu one; it's the same key).
2. **Stop hard-coding `bookworm`.** Use `{{ ansible_distribution_release }}`
   in the Docker repo line, so the role follows whatever release is
   installed. `apt_repository` writes a new file, so remove the old
   `download_docker_com_linux_debian.list` in the same change, to avoid
   having the repo listed twice.
3. Optional: manage the Tailscale repo the same way, so all third-party
   repos come from Ansible.
4. Deploy with `--tags` for the docker role's base tasks, then check that
   `apt update` is clean and nothing changed on bookworm.

## Step 2: backups (the day of the upgrade)

1. **Take a netcup snapshot of the whole server** from the netcup control
   panel. This is the rollback plan. Check that the snapshot has finished
   before going on.
2. Trigger the existing S3 backups by hand (Paperless, GoToSocial), as a
   second copy of the most important data.
3. Note the running containers (`docker ps`) and the dockcheck output, to
   compare against afterwards.

## Step 3: the upgrade (allow about an hour of downtime)

Run in `tmux` over Tailscale SSH, so a dropped connection doesn't kill
apt. Keep the netcup VNC console open as a fallback.

1. Bring bookworm fully up to date: `apt update && apt full-upgrade`,
   reboot if the kernel changed, then `apt autoremove --purge`.
2. Stop the containers, so databases aren't written to while their
   libraries and Docker are replaced: `systemctl stop docker docker.socket`.
3. Change `bookworm` to `trixie` in `/etc/apt/sources.list` (including
   `trixie-security` and `trixie-updates`) and in the Tailscale list. The
   Docker list comes from step 1.
4. `apt update`, then the minimal upgrade first:
   `apt upgrade --without-new-pkgs`, then `apt full-upgrade`.
   - Keep our versions of config files Ansible manages (sshd drop-in,
     `20auto-upgrades`). For anything else, look at the diff before
     choosing.
5. Reboot into kernel 6.12.

## Step 4: checks after the reboot

- `cat /etc/debian_version` shows 13.x; `uname -r` shows 6.12.
- SSH key login works over both the public IP and Tailscale; passwords
  are still refused (`sshd -T | grep -i password`).
- `tailscale status` is up; `ufw status` still has the four Docker
  forward rules (Caddy 80/443, Gitea SSH).
- Docker is running and every container came back (`docker ps` against the
  list from step 2). Check the Postgres and MariaDB/Mongo containers' logs.
- Public sites load through Caddy; ports 22, 80, 443 and 222 are the only
  ones open from outside.
- `/tmp`: trixie can move `/tmp` into RAM (tmpfs) and clean `/tmp` and
  `/var/tmp` on a timer. Check `findmnt /tmp` and make sure nothing here
  keeps files there long-term.
- `unattended-upgrades` picks up `trixie-security` (dry run:
  `unattended-upgrade --dry-run -d`).
- Run `bin/deploy --check` against the whole site to confirm Ansible
  still works with the new Python (3.13) and reports no unexpected changes.
- Then `apt autoremove --purge`, and purge the `rc` kernel packages.

## Rollback

If the server doesn't come back or something important is broken and
can't be fixed quickly, restore the netcup snapshot from step 2. Once
things are stable, delete the snapshot (netcup snapshots slow down the
disk over time).

## Open questions

- Does the netcup plan include snapshots, and how big a snapshot can it
  take? (The server uses 246 GB.)
- When is a good maintenance window?