diff --git a/CLAUDE.local.md b/CLAUDE.local.md index 9c581e0..0ba20e0 100644 --- a/CLAUDE.local.md +++ b/CLAUDE.local.md @@ -4,4 +4,4 @@ - Whenever I talk about cron jobs I am referring to cron jobs on the remote servers managed by ansible, never the local machine - never use secrets.enc - all secrets go in vault.yml, never secrets.enc, never some random file you want to create, only ever vault.yml. you decrypt vault.yml with vault_pass -- Never use ansible-vault edit. always decrypt, make the changes, then encrypt \ No newline at end of file +- Never use ansible-vault edit. Edit vault.yml only with bin/vault-run (it decrypts, runs your edit, checks and re-encrypts) \ No newline at end of file diff --git a/CLAUDE.md b/CLAUDE.md index 585ca50..23be735 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,9 +4,13 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Overview -This is a personal infrastructure Ansible playbook that automates deployment and management of 21 self-hosted Docker services across two domains (`thesatelliteoflove.com` and `nerder.land`). The setup uses Tailscale VPN for secure networking and Caddy for reverse proxy with automated HTTPS. +Personal infrastructure as Ansible: 24 self-hosted Docker services on one +server, across `thesatelliteoflove.com` and `nerder.land`. Caddy is the +reverse proxy (automatic HTTPS), Pocket ID is the passkey SSO, and Tailscale +is how Ansible reaches the server. See `README.md` for the service list. -**Important**: Always review `DEPLOYMENT_LEARNINGS.md` when working on this repository for lessons learned and troubleshooting guidance. +If a local `DEPLOYMENT_LEARNINGS.md` exists (it's gitignored), read it for +past lessons. ## Working alongside other agents (read first) @@ -41,156 +45,123 @@ other. Follow these steps: ## Common Commands -### Initial Setup ```bash -# Install Ansible dependencies -ansible-galaxy install -r requirements.yml +bin/worktree # start a task (see above) +bin/deploy --tags # deploy one service +bin/deploy --tags media,productivity # deploy categories +bin/deploy --tags services # every service, no system setup +bin/deploy --tags common,cron # base setup and cron jobs only +bin/deploy --check --diff --tags # dry run, allowed from any checkout +PLAYBOOK=dns.yml bin/deploy # Route53 records +bin/vault-run # edit secrets ($VAULT_PLAIN) -# Bootstrap new server (creates user, installs Tailscale, security hardening) -ansible-playbook bootstrap.yml -i hosts.yml - -# Deploy all Docker services -ansible-playbook site.yml -i hosts.yml - -# Update DNS records in AWS Route53 -ansible-playbook dns.yml -i hosts.yml +# First-time setup of a new server (public IP, before Tailscale) +.venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass ``` -### Service Management -```bash -# Deploy specific services using tags (now properly isolated) -ansible-playbook site.yml -i hosts.yml --tags caddy --vault-password-file vault_pass -ansible-playbook site.yml -i hosts.yml --tags pocketid --vault-password-file vault_pass -ansible-playbook site.yml -i hosts.yml --tags gitea --vault-password-file vault_pass -ansible-playbook site.yml -i hosts.yml --tags docker --vault-password-file vault_pass # entire docker role -ansible-playbook site.yml -i hosts.yml --tags services --vault-password-file vault_pass # all services only - -# Deploy services by category -ansible-playbook site.yml -i hosts.yml --tags infrastructure --vault-password-file vault_pass -ansible-playbook site.yml -i hosts.yml --tags media,productivity --vault-password-file vault_pass -ansible-playbook site.yml -i hosts.yml --tags development,monitoring --vault-password-file vault_pass - -# Deploy only infrastructure components -ansible-playbook site.yml -i hosts.yml --tags common,cron --vault-password-file vault_pass -``` +The venv (`.venv/`), `vault_pass` and `group_vars/all/vault.yml` live in the +main checkout; the `bin/` helpers find them from any worktree. ## Architecture -### Host Configuration -- **Bootstrap Host** (`netcup`): 152.53.36.98 - Initial server setup target -- **Docker Host** (`docker-01`): 100.70.169.99 - Main service deployment via Tailscale +### Host +One server, `cogitator-01` (netcup VPS), listed twice in `hosts.yml`: +- `netcup` (group `bootstrap`): public IP 152.53.36.98, used only by `bootstrap.yml`. +- `docker-01` (group `docker`): Tailscale IP 100.70.169.99, used by `site.yml`. -### Role Structure -- **bootstrap**: Initial server hardening, user creation, Tailscale VPN setup -- **common**: Basic system configuration, UFW firewall management -- **docker**: Comprehensive service deployment (21 containerized applications, organized by category) -- **cron**: Scheduled task management (currently Warhammer RSS feed generation) +DNS A records point at the public IP; Caddy on the same machine routes by +subdomain. -### Docker Role Organization (Reorganized into Logical Categories) -The docker role is now organized into logical service groups under `roles/docker/tasks/`: +### Roles (`site.yml` runs common, docker, cron) +- **bootstrap**: user, SSH key, passwordless sudo, UFW, then `artis3n.tailscale`. +- **common**: `aptitude`, `jq`, UFW on with default deny. +- **docker**: Docker CE, shell setup (zsh, Atuin, bat), ufw-docker, the `lava` + network, and every service. +- **cron**: dockcheck (08:00, auto-updates labelled containers, Gotify notices) + and the Warhammer RSS feed (09:10). -- **infrastructure/**: Core platform components - - Caddy (reverse proxy), Pocket ID (passkey SSO), Dockge (container management) -- **development/**: Development and collaboration tools - - Gitea, ByteStash -- **media/**: Content creation and consumption - - Audiobookshelf, Kavita, Karakeep (formerly Hoarder), Manyfold -- **productivity/**: Personal organization and document management - - Paperless-NGX, Baikal (CalDAV/CardDAV), Syncthing, Heyform, Dawarich, Obsidian LiveSync -- **communication/**: Social media - - GoToSocial (Fediverse) -- **monitoring/**: System monitoring and alerts - - Changedetection, AppriseAPI, Gotify, Cronmaster +### Docker role categories (`roles/docker/tasks//`) +- **infrastructure/**: Caddy, Pocket ID, Gondulf (IndieAuth), Dockge +- **development/**: Gitea (+ act_runner), ByteStash +- **media/**: Audiobookshelf, Kavita, Karakeep (files named `hoarder`), Manyfold +- **productivity/**: Paperless-ngx, Baikal, Syncthing, HeyForm, Dawarich, Obsidian LiveSync, Gokapi +- **communication/**: GoToSocial, Listmonk, StarPunk (Micropub) +- **monitoring/**: Changedetection, Apprise API, Gotify (+ iGotify), Cronmaster -### Variable Management -**Critical**: This infrastructure uses a centralized variable hierarchy in `group_vars/all/`: +Each category has a tag (`infrastructure`, `media`, …) and each service its +own tag; `services` covers all of them. Karakeep has both `karakeep` and +`hoarder` tags. -- **domains.yml**: Domain and subdomain mappings (use `{{ subdomains.service }}`) -- **infrastructure.yml**: Network configuration, Docker settings (`{{ docker.network_name }}`, `{{ docker.hairpin_ip }}`) -- **vault.yml**: Encrypted secrets with `vault_` prefix -- **services.yml**: Service-specific configuration and feature flags +### Variables (`group_vars/all/`) +- **domains.yml**: `primary_domain`, `secondary_domain`, `subdomains.`, email domains. +- **infrastructure.yml**: `docker.*` (network name, subnet, fixed IPs), `smtp.*` (Resend), `network.*` (public and Tailscale IPs). +- **vault.yml**: encrypted secrets, one `vault_` dict per service. -**Important**: All templates use variables instead of hardcoded values. Never hardcode domains, IPs, or secrets. +Never hardcode domains, IPs or secrets in templates; use these variables. -### Data Structure -- All service data stored in `/opt/stacks/[service-name]/` on docker host -- Docker Compose files generated from Jinja2 templates in `roles/docker/templates/` -- Environment files templated for services requiring configuration -- All configurations use centralized variables for consistency +### Data +- Each service lives in `/opt/stacks//` on the server (Dockge in `/opt/dockge/`). +- Compose files come from `roles/docker/templates/-compose.yml.j2`; env and config files from `-env.j2` / `.env.j2` / `-config.json.j2`. +- Retired services' final backups are in `/opt/stacks/_retired/`. ## Key Implementation Details -### Template-Driven Configuration -The docker role uses Jinja2 templates exclusively for all services. When modifying services: -- Update templates in `roles/docker/templates/[service]-compose.yml.j2` -- Environment files use `.env.j2` templates where needed -- Task files organized by category in `roles/docker/tasks/[category]/[service].yml` -- All services now use templated configurations (no static compose files) +### Adding or changing a service +- Template: `roles/docker/templates/-compose.yml.j2`. +- Tasks: `roles/docker/tasks//.yml`, imported with a tag in that category's `main.yml`. +- Public access: subdomain in `domains.yml`, route in `Caddyfile.j2`, A record in `dns.yml`. +- Use `diff: false` / `no_log: true` on tasks that write secrets. +- Add the label `mag37.dockcheck.update: true` if dockcheck may update it unattended. -### DNS Management -The `dns.yml` playbook manages AWS Route53 records for both domains. All subdomains point to the netcup server (152.53.36.98), with Caddy handling internal routing to the docker host via Tailscale. +### DNS +`dns.yml` manages Route53 A records for both domains with a Route53-only IAM +user (`vault_aws_dns`). It also deletes the records listed under +`removed_records`; add a retired service's subdomain there. -### Security Architecture -- Tailscale provides secure networking between management and service hosts -- Services are network-isolated using Docker -- Caddy handles SSL termination with automatic Let's Encrypt certificates -- UFW firewall managed through Docker integration script +### Security +- Caddy terminates TLS with Let's Encrypt. +- `ufw-docker` makes UFW govern container ports. Only Caddy (80/443) and + Gitea SSH (host port 222) are allowed in from the internet. +- Admin UIs (Syncthing, Cronmaster, Apprise, CouchDB) bind to the Tailscale IP only. -### Service Dependencies -Most services log in through Pocket ID (OIDC, passkeys) at `{{ subdomains.pocketid }}`. When deploying new services, consider: -- Whether SSO integration is needed -- Caddy routing configuration for subdomain access -- Network connectivity requirements within Docker stack -- Hairpinning fixes for internal service-to-service communication +### SSO (Pocket ID) +Pocket ID (`id.thesatelliteoflove.com`) replaced Authentik on 2026-10-31. +Apps that log in with it: Gitea, ByteStash, Audiobookshelf, Kavita, Karakeep, +Manyfold, Paperless-ngx, Gokapi, GoToSocial. Some read the client from env +vars; Gitea, Audiobookshelf and Kavita store it in their own database (set in +their admin UI). OIDC clients are created in Pocket ID's UI and their ID and +secret go in the vault as `vault_.pocketid`. StarPunk logs in with +IndieAuth through Gondulf instead. ### Shared Docker Network (`lava`) Created by the docker role as `{{ docker.subnet }}` with `ip_range: {{ docker.ip_range }}`. Docker only hands out dynamic addresses from that range, so `172.20.0.x` is reserved for fixed IPs: Caddy (`docker.hairpin_ip`, .5) and Gitea (`docker.gitea_ip`, .6). Both need fixed IPs because ufw-docker firewall rules point at container IPs. A network without the range (the 2024 original) is migrated once with `migrate-lava-network.yml`. -### Hairpinning Resolution -Services inside Docker containers cannot reach external domains that resolve to the same server. Fix by adding `extra_hosts` mappings: +### Hairpinning +Containers can't reach domains that resolve to this server's public IP. Map +any of our domains a container calls to Caddy with `extra_hosts`: ```yaml extra_hosts: - "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}" - - "{{ subdomains.cal }}:{{ docker.hairpin_ip }}" ``` -Common domains requiring hairpinning fixes: -- `{{ subdomains.pocketid }}` (Pocket ID SSO) -- `{{ subdomains.cal }}` (Baikal CalDAV) -- Any service domain the container needs to communicate with +Most often needed for `subdomains.pocketid` (OIDC); also `subdomains.git` +(Gitea itself), `subdomains.gondulf` and `primary_domain` (StarPunk). -**Note**: Use variables instead of hardcoded values for maintainability. +### Template pattern +```yaml +networks: + default: + external: true + name: "{{ docker.network_name }}" +``` -### Service-Specific Reference Configurations -- **Dawarich**: Based on production compose file at https://github.com/Freika/dawarich/blob/master/docker/docker-compose.production.yml +### Service-Specific Notes +- **Dawarich**: based on https://github.com/Freika/dawarich/blob/master/docker/docker-compose.production.yml +- **Gokapi**: `config.json` is only written on a fresh install (`force: false`); delete it on the server to apply template changes. +- **Caddy**: Caddyfile changes restart it via a handler; the stack uses `recreate: never`. ## Service Memories - karakeep (formerly called hoarder) is deployed with both 'hoarder' and 'karakeep' tags for backward compatibility - whenever i ask you what containers need updates, run dockcheck and return a list of containers needing updates - when i ask for the status container updates i want you to run dockcheck on the docker host https://github.com/mag37/dockcheck?ref=selfh.st - -## Variable Management Implementation Notes -**Major Infrastructure Update**: Variable management system was implemented to replace all hardcoded values with centralized variables. - -### Key Changes Made: -- Created comprehensive `group_vars/all/` structure -- Updated all Docker Compose templates to use variables -- Fixed service tag isolation (individual service tags now deploy only that service) -- Standardized domain and network configuration -- Organized secrets by service with consistent `vault_` prefix - -### Service Tag Fix: -**Critical**: Service tags are properly isolated. Individual service tags deploy only that service. The `services` tag runs all service tasks across all categories. - -### Template Pattern: -All templates now follow this pattern: -```yaml -# Use variables, not hardcoded values -networks: - default: - external: true - name: "{{ docker.network_name }}" -extra_hosts: - - "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}" -``` \ No newline at end of file diff --git a/README.md b/README.md index 893b81a..b683a5f 100644 --- a/README.md +++ b/README.md @@ -1,157 +1,154 @@ # Personal Infrastructure Ansible Playbook -This Ansible playbook automates the setup and management of a personal self-hosted infrastructure running Docker containers for various services. +Ansible that builds and runs a personal self-hosted stack: 24 Docker services +on one server, served on `thesatelliteoflove.com` and `nerder.land`, with +Caddy for HTTPS and Pocket ID (passkeys) for single sign-on. -## Overview +## The server -The playbook manages two main environments: -- **Bootstrap server** (`netcup`): Initial server setup with Tailscale VPN -- **Docker server** (`docker-01`): Main application server running containerized services +There is one server (hostname `cogitator-01`, a netcup VPS). The inventory +lists it twice, under two names: -## Services Deployed +| Inventory name | Address | Used by | +|---|---|---| +| `netcup` (group `bootstrap`) | 152.53.36.98 (public IP) | `bootstrap.yml`, for first-time setup before Tailscale is up | +| `docker-01` (group `docker`) | 100.70.169.99 (Tailscale IP) | `site.yml`, for everything after that | -The Docker role deploys and manages 21 self-hosted services organized into logical categories: +All public DNS records point at the public IP. Caddy on the same machine +takes ports 80/443 and passes each subdomain to the right container. -### Infrastructure -- **Caddy** (Reverse proxy with automatic HTTPS) -- **Pocket ID** (passkey SSO/Identity Provider) -- **Dockge** (Container management) +## Services -### Development -- **Gitea** (Git repository hosting) -- **ByteStash** (Code snippet management) +| Category | Service | Address | Login | +|---|---|---|---| +| Infrastructure | Caddy (reverse proxy, static sites) | — | — | +| | Pocket ID (passkey OIDC provider) | id.thesatelliteoflove.com | passkeys | +| | Gondulf (IndieAuth for the main site) | gondulf.thesatelliteoflove.com | email code | +| | Dockge (compose stack manager) | port 5001, no public route | local | +| Development | Gitea + Actions runner | git.thesatelliteoflove.com (SSH on port 222) | Pocket ID | +| | ByteStash (code snippets) | snippets.thesatelliteoflove.com | Pocket ID | +| Media | Audiobookshelf | audio.thesatelliteoflove.com | Pocket ID | +| | Kavita (ebooks, OPDS, KOReader sync) | books.thesatelliteoflove.com | Pocket ID | +| | Karakeep (bookmarks, formerly Hoarder) | bookmarks.thesatelliteoflove.com | Pocket ID | +| | Manyfold (3D models) | models.thesatelliteoflove.com | Pocket ID | +| Productivity | Paperless-ngx (documents) | paper.thesatelliteoflove.com | Pocket ID | +| | Baikal (CalDAV/CardDAV) | cal.thesatelliteoflove.com | local | +| | Syncthing | Tailscale only, port 8384 | local | +| | HeyForm (forms) | forms.nerder.land | local | +| | Dawarich (location history) | loclog.thesatelliteoflove.com | local | +| | Obsidian LiveSync (CouchDB) | Tailscale only, port 5984 | local | +| | Gokapi (file sharing) | gokapi.thesatelliteoflove.com | Pocket ID | +| Communication | GoToSocial (fediverse) | social.thesatelliteoflove.com | Pocket ID | +| | Listmonk (mailing lists) | listmonk.nerder.land | local | +| | StarPunk (Micropub for the main site) | starpunk.thesatelliteoflove.com | IndieAuth | +| Monitoring | Changedetection | watcher.thesatelliteoflove.com | local | +| | Apprise API (notifications) | Tailscale only, port 8000 | — | +| | Gotify + iGotify assistant | gotify.thesatelliteoflove.com, gotify-assistant.thesatelliteoflove.com | local | +| | Cronmaster (cron UI) | Tailscale only, port 40123 | local | -### Media -- **Audiobookshelf** (Audiobook server) -- **Kavita** (E-book server) -- **Karakeep** (Bookmark manager, formerly Hoarder) -- **Manyfold** (3D model organizer) +"Tailscale only" means the port is bound to the Tailscale IP and has no public +address. -### Productivity -- **Paperless-NGX** (Document management) -- **Baikal** (CalDAV/CardDAV server) -- **Syncthing** (File synchronization) -- **HeyForm** (Form builder) -- **Dawarich** (Location tracking) -- **Obsidian LiveSync** (Note synchronization) +Caddy also serves static sites from `/opt/stacks/caddy/site/` (the files are +not managed by Ansible): the apex `thesatelliteoflove.com` (plus WebFinger for +the fediverse and Tailscale login), `pdg.thesatelliteoflove.com`, +`nerder.land` and `repair.nerder.land`. `repaircafe.nerder.land` redirects to +the Listmonk signup form. -### Communication -- **GoToSocial** (Fediverse/Mastodon) - -### Monitoring -- **Changedetection** (Website change monitoring) -- **AppriseAPI** (Notification service) -- **Gotify** (Push notifications) -- **Cronmaster** (Scheduled container tasks) - -## Structure +## Layout ``` -├── site.yml # Main playbook -├── bootstrap.yml # Server bootstrap playbook -├── dns.yml # AWS Route53 DNS management -├── hosts.yml # Inventory file -├── requirements.yml # External role dependencies +├── site.yml # Main playbook: common, docker and cron roles +├── bootstrap.yml # First-time server setup (user, SSH key, UFW, Tailscale) +├── dns.yml # Route53 A records (adds current ones, removes retired ones) +├── deploy-guard.yml # Stops real runs from anything but a clean origin/main +├── migrate-lava-network.yml # One-off: recreate the shared docker network +├── hosts.yml # Inventory +├── requirements.yml # Galaxy roles (artis3n.tailscale) +├── bin/ +│ ├── worktree # Start a task in its own git worktree +│ ├── deploy # Run a playbook while holding the deploy lock +│ └── vault-run # Edit vault.yml safely +├── group_vars/all/ +│ ├── domains.yml # Domains and subdomain map +│ ├── infrastructure.yml # Docker network, IPs, SMTP, paths +│ └── vault.yml # Encrypted secrets (gitignored) └── roles/ - ├── bootstrap/ # Initial server setup - ├── common/ # Common server configuration - ├── cron/ # Scheduled tasks - └── docker/ # Docker services deployment + ├── bootstrap/ # First-time setup + ├── common/ # Base packages and UFW + ├── docker/ # Docker, shell setup and every service + └── cron/ # Host cron jobs (dockcheck, Warhammer feed) ``` -## Roles Documentation - -Each role has detailed documentation in its respective directory: - -### [Bootstrap Role](roles/bootstrap/README.md) -Performs initial server setup and hardening: -- Creates user accounts with SSH key authentication -- Configures passwordless sudo and security hardening -- Installs essential packages and configures UFW firewall -- Sets up Tailscale VPN for secure network access - -### [Common Role](roles/common/README.md) -Provides shared configuration for all servers: -- Installs common packages (aptitude) -- Enables UFW firewall with default deny policy -- Ensures consistent base configuration across infrastructure - -### [Cron Role](roles/cron/README.md) -Manages scheduled tasks and automation: -- **Warhammer RSS Feed Updater**: Daily job that generates and updates RSS feeds -- Integrates with Docker services for content generation -- Supports easy addition of new scheduled tasks - -### [Docker Role](roles/docker/README.md) -The most comprehensive role, deploying 21 containerized services organized into logical categories: -- **Infrastructure**: Caddy reverse proxy, Pocket ID SSO, Dockge management -- **Development**: Gitea, ByteStash -- **Media**: Audiobookshelf, Kavita, Karakeep, Manyfold -- **Productivity**: Paperless-NGX, Baikal, Syncthing, HeyForm, Dawarich, Obsidian LiveSync -- **Communication**: GoToSocial -- **Monitoring**: Changedetection, AppriseAPI, Gotify, Cronmaster -- **Template-Driven**: All services use Jinja2 templates for consistent configuration -- **Category-Based Deployment**: Deploy services by category using Ansible tags +Role details: [bootstrap](roles/bootstrap/README.md), +[common](roles/common/README.md), [docker](roles/docker/README.md), +[cron](roles/cron/README.md). ## Usage -### Prerequisites +### Setup -1. Install Ansible and required collections: - ```bash - ansible-galaxy install -r requirements.yml - ``` - -2. Configure your inventory in `hosts.yml` with your server details - -### Bootstrap a New Server +The `bin/` helpers use the virtualenv at `.venv/` in the main checkout. The +`ansible` package brings the collections used here (`community.docker`, +`community.general`, `amazon.aws`, `ansible.posix`); `boto3` is needed for +Route53. ```bash -ansible-playbook bootstrap.yml -i hosts.yml +python -m venv .venv && .venv/bin/pip install ansible boto3 +.venv/bin/ansible-galaxy install -r requirements.yml ``` -This will: -- Create a user account -- Install and configure Tailscale VPN -- Set up basic security +`vault_pass` and `group_vars/all/vault.yml` are gitignored and live only in +the main checkout. `bin/worktree` links them into each worktree. -### Deploy Docker Services +### Making a change + +Several agents work on this repo at once, so every change follows the same +steps (see [CLAUDE.md](CLAUDE.md) for why): ```bash -ansible-playbook site.yml -i hosts.yml +bin/worktree my-change # new worktree off origin/main +# ...edit, commit... +git fetch && git rebase origin/main +git push origin HEAD:main +bin/deploy --tags # deploy that exact commit ``` -Deploy specific services using tags: -```bash -# Deploy by service category -ansible-playbook site.yml -i hosts.yml --tags infrastructure -ansible-playbook site.yml -i hosts.yml --tags media,productivity - -# Deploy all services (skips system setup) -ansible-playbook site.yml -i hosts.yml --tags services - -# Deploy individual services -ansible-playbook site.yml -i hosts.yml --tags caddy -ansible-playbook site.yml -i hosts.yml --tags pocketid -``` - -### Manage DNS Records +### Deploying ```bash -ansible-playbook dns.yml -i hosts.yml +bin/deploy --tags kavita # one service +bin/deploy --tags media,productivity # whole categories +bin/deploy --tags services # every service, no system setup +bin/deploy # everything in site.yml +bin/deploy --check --diff --tags caddy # dry run (allowed from any checkout) +PLAYBOOK=dns.yml bin/deploy # DNS records ``` -Updates AWS Route53 DNS records for configured domains (`thesatelliteoflove.com` and `nerder.land`). +Category tags: `infrastructure`, `development`, `media`, `productivity`, +`communication`, `monitoring`. Role tags: `common`, `docker`, `cron`. Each +service has its own tag, named after its task file (e.g. `paperlessngx`, +`obsidian-livesync`); Karakeep answers to both `karakeep` and `hoarder`. -## Configuration +### Bootstrapping a new server -- Service configurations are templated in `roles/docker/templates/` -- Environment variables and secrets should be managed through Ansible Vault -- Docker Compose files are generated from Jinja2 templates +```bash +.venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass +``` -## Security Notes +### Secrets -- Uses Tailscale for secure network access -- Caddy provides automatic HTTPS with Let's Encrypt -- Services are containerized for isolation -- UFW firewall rules are managed via Docker integration \ No newline at end of file +All secrets live in `group_vars/all/vault.yml` under `vault_` keys. +Edit them only with `bin/vault-run `, never `ansible-vault edit`. + +## Security notes + +- UFW allows SSH from anywhere. `ufw-docker` blocks container ports from the + internet except Caddy (80/443) and Gitea SSH (222). +- Admin tools (Syncthing, Cronmaster, Apprise, CouchDB) bind to the Tailscale + IP only. Dockge has no public route. +- Pocket ID is the single sign-on for most apps. Authentik was retired on + 2026-10-31. +- Paperless-ngx and GoToSocial back up their volumes daily to S3 (7-day + retention). Other services have no automated backups yet (see + [todo.md](todo.md)). diff --git a/roles/bootstrap/README.md b/roles/bootstrap/README.md index 1f70cfd..5756357 100644 --- a/roles/bootstrap/README.md +++ b/roles/bootstrap/README.md @@ -1,41 +1,29 @@ # Bootstrap Role -## Purpose -Performs initial server setup and hardening for new Ubuntu/Debian servers. +First-time setup for a fresh Debian/Ubuntu server. Run once, against the +server's public IP (inventory group `bootstrap`, host `netcup`), before +`site.yml` can reach it over Tailscale. -## What It Does +## What it does +- Installs `aptitude`, `curl`, `vim`, `git` and `ufw`. +- Gives the `sudo` group passwordless sudo. +- Creates the user `created_username` (set to `phil` in `bootstrap.yml`) in + the `sudo` group, with your local `~/.ssh/id_ed25519.pub` as its SSH key. +- Sets `PermitRootLogin prohibit-password` (root can still log in with a key). +- UFW: allows OpenSSH, enables the firewall with default deny. -### User Management -- Creates a new user account with sudo privileges (specified by `created_username` variable) -- Configures passwordless sudo for the sudo group -- Sets up SSH key authentication using your local `~/.ssh/id_ed25519.pub` key -- Disables root password authentication +`bootstrap.yml` then runs the `artis3n.tailscale` role to join the tailnet, +using `vault_infrastructure.tailscale_key` from the vault. -### System Packages -- Installs `aptitude` for better package management -- Installs essential packages: - - `curl` - HTTP client - - `vim` - Text editor - - `git` - Version control - - `ufw` - Uncomplicated Firewall - -### Security Configuration -- Configures UFW firewall to: - - Allow SSH connections - - Enable firewall with default deny policy -- Hardens SSH configuration - -## Variables Required -- `created_username`: The username to create (typically set in bootstrap.yml) -- `tailscale_key`: Tailscale authentication key (prompted during playbook run) - -## Dependencies -- Requires the `artis3n.tailscale` role for VPN setup -- Requires your SSH public key at `~/.ssh/id_ed25519.pub` +## Requirements +- `ansible-galaxy install -r requirements.yml` (for `artis3n.tailscale`). +- `~/.ssh/id_ed25519.pub` on the machine running Ansible. +- Root (or another sudo user) SSH access to the new server. ## Usage ```bash -ansible-playbook bootstrap.yml -i hosts.yml +.venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass ``` -This role is designed to be run once on a fresh server before deploying other services. \ No newline at end of file +Once the server is on Tailscale, put its Tailscale IP in `hosts.yml` under +`docker` and use `bin/deploy` from then on. diff --git a/roles/common/README.md b/roles/common/README.md index ece64b7..bf1a2da 100644 --- a/roles/common/README.md +++ b/roles/common/README.md @@ -1,23 +1,13 @@ # Common Role -## Purpose -Provides shared configuration and security setup that applies to all servers in the infrastructure. +Base setup applied first by `site.yml` (tag `common`). -## What It Does - -### System Packages -- Installs `aptitude` for better package management and dependency resolution -- Updates package cache to ensure latest package information - -### Security Configuration -- Enables UFW (Uncomplicated Firewall) with default deny policy -- Provides baseline firewall protection for all managed servers +## What it does +- Installs `aptitude` and `jq`. +- Makes sure UFW is enabled with default deny. The SSH rule comes from the + bootstrap role, and container rules from `ufw-docker` in the docker role. ## Usage -This role is automatically applied to all servers in the infrastructure as part of the main site.yml playbook. It ensures consistent base configuration across all managed systems. - -## Dependencies -None - this is a foundational role that other roles can depend on. - -## Notes -This role is designed to be lightweight and provide only the most essential common functionality. Server-specific configurations should be handled by dedicated roles like `docker` or `bootstrap`. \ No newline at end of file +```bash +bin/deploy --tags common +``` diff --git a/roles/cron/README.md b/roles/cron/README.md index 1241030..3d9b9d8 100644 --- a/roles/cron/README.md +++ b/roles/cron/README.md @@ -1,35 +1,36 @@ # Cron Role -## Purpose -Manages scheduled tasks and automated maintenance jobs for the infrastructure. +Cron jobs on the server (tag `cron`). "Cron jobs" in this repo always means +jobs on the server, never on the machine running Ansible. -## What It Does +## Jobs -### Warhammer RSS Feed Updater -- Copies `update_warhammer_feed.sh` script to `/usr/local/bin/` with executable permissions -- Creates a daily cron job that runs at 09:10 AM -- The script performs these actions: - 1. Creates a temporary directory `/tmp/warhammer_feed` - 2. Runs a custom Docker container (`git.thesatelliteoflove.com/phil/rss-warhammer`) to generate RSS feed - 3. Copies the generated `warhammer_rss_feed.xml` to `/opt/stacks/caddy/site/tsol/feeds/` +| Time | User | Job | +|---|---|---| +| 08:00 daily | phil | `~/.local/bin/run_dockcheck.sh`: container updates | +| 09:10 daily | root | `/usr/local/bin/update_warhammer_feed.sh`: Warhammer RSS feed | -## Files Managed -- `/usr/local/bin/update_warhammer_feed.sh` - RSS feed update script -- Cron job: "Update Warhammer RSS Feed" (daily at 09:10) - -## Dependencies -- Requires Docker to be installed and running -- Depends on the following Docker stacks being deployed: - - Custom RSS generator container at `git.thesatelliteoflove.com/phil/rss-warhammer` - - Caddy web server stack at `/opt/stacks/caddy/` - -## Usage -This role is automatically applied as part of the main site.yml playbook with the `cron` tag. +### Dockcheck (container updates) +- Downloads [dockcheck](https://github.com/mag37/dockcheck) and its Gotify + notify templates into `/home/phil/.local/bin/`. +- Templates `/home/phil/.config/dockcheck.config` from `dockcheck.config.j2` + (Gotify token from `vault_dockcheck`). +- Settings: `OnlyLabel=true` and `AutoMode=true`, so containers labelled + `mag37.dockcheck.update=true` are pulled and recreated automatically; any + other container with an update gets a Gotify notice. `dawarich_db` and + `dawarich_redis` are skipped. +- `run_dockcheck.sh` logs to `/var/log/dockcheck/dockcheck.log`. +To see what needs updating by hand, run dockcheck on the server: ```bash -# Deploy only cron jobs -ansible-playbook site.yml -i hosts.yml --tags cron +ssh phil@100.70.169.99 '~/.local/bin/dockcheck.sh -n' ``` -## Customization -To add additional cron jobs, create new tasks in the main.yml file following the same pattern as the Warhammer feed updater. \ No newline at end of file +### Warhammer RSS feed +`update_warhammer_feed.sh` runs the `git.thesatelliteoflove.com/phil/rss-warhammer` +container to build `warhammer_rss_feed.xml` and copies it to +`/opt/stacks/caddy/site/tsol/feeds/`, where Caddy serves it on the main site. + +## Adding a job +Add a `cron` task to `tasks/main.yml` (copy the script with `copy` first if it +needs one), then `bin/deploy --tags cron`. diff --git a/roles/docker/README.md b/roles/docker/README.md index 4d3a00b..ef06ded 100644 --- a/roles/docker/README.md +++ b/roles/docker/README.md @@ -1,221 +1,144 @@ # Docker Role -## Purpose -Deploys and manages a comprehensive self-hosted infrastructure with 24 containerized services organized into logical categories, transforming a server into a personal cloud platform with authentication, media management, productivity tools, and development services. +Installs Docker and deploys all 24 services. Each service is a Docker Compose +stack in `/opt/stacks//` (Dockge lives in `/opt/dockge/`), built from +Jinja2 templates in this role. -## Architecture Overview +## What runs before the services -### Network Configuration -- **External Network**: All services connect to shared Docker network (configurable) -- **Reverse Proxy**: Caddy handles all ingress traffic with automatic HTTPS -- **Service Discovery**: Container-to-container communication using service names -- **Firewall Integration**: UFW-Docker script properly configures firewall rules +`tasks/main.yml` does the system setup, then imports each category: -### Security Features -- **Centralized SSO**: Pocket ID provides passkey OIDC authentication for most services -- **Network Isolation**: Services restricted to appropriate network segments -- **Container Hardening**: Non-root users, capability dropping, security options -- **Secret Management**: Ansible vault for sensitive configuration -- **Variable Management**: Centralized variable hierarchy using group_vars structure +1. Base packages (`python3-pip`, `virtualenv`, `duf`, …). +2. Shell setup (`shell.yml`): zsh as the login shell, Atuin (installed and + logged in once, dotfile sync on), and `bat`. +3. Docker CE from Docker's apt repository. +4. `ufw-docker.sh` copied to `/usr/local/bin/` and its rules installed (UFW is + restarted once after install). +5. The shared `lava` network (see below). +6. `/opt/stacks` and `/opt/dockge`. -## Services Deployed (Organized by Category) +## Services by category -### Infrastructure (`infrastructure/`) -- **Caddy** - Reverse proxy with automatic HTTPS (static IP: 172.20.0.5) -- **Pocket ID** - Passkey OIDC provider -- **Dockge** - Docker compose stack management UI +Tags: each category file has a category tag, each service its own tag. -### Development (`development/`) -- **Gitea** - Self-hosted Git with CI/CD runners -- **Code Server** - VS Code in the browser -- **ByteStash** - Code snippet management and organization +### Infrastructure (`infrastructure`) +| Tag | Service | Notes | +|---|---|---| +| `caddy` | Caddy | Fixed IP 172.20.0.5. Ports 80/443 (+443/udp). `Caddyfile.j2` changes restart it via a handler; the stack itself is deployed with `recreate: never`. Static sites are served from `/opt/stacks/caddy/site/`. | +| `pocketid` | Pocket ID v2 | Passkey OIDC provider at `id.`. OIDC clients are created in its web UI; their IDs and secrets go in the vault. | +| `gondulf` | Gondulf | IndieAuth server for the main site (own image from Gitea). Sends login codes by email through Resend. | +| `dockge` | Dockge | Compose UI on port 5001, managing `/opt/stacks`. | -### Media (`media/`) -- **Audiobookshelf** - Audiobook server (Pocket ID login) -- **Kavita** - E-book server (Pocket ID login) -- **Ghost** - Modern blogging platform -- **Pinchflat** - YouTube video archiving -- **Pinry** - Pinterest-like image board -- **Karakeep** - Bookmark management with AI tagging -- **Manyfold** - 3D model file organization +### Development (`development`) +| Tag | Service | Notes | +|---|---|---| +| `gitea` | Gitea + act_runner | Fixed IP 172.20.0.6 so the ufw-docker rule for SSH (host port 222) stays valid. Pocket ID login is set up in Gitea's admin UI; OpenID 2.0 login is off. | +| `bytestash` | ByteStash | Pocket ID through env vars. | -### Productivity (`productivity/`) -- **Paperless-ngx** - Document management with OCR -- **MMDL** - Task and calendar management with CalDAV integration -- **Baikal** - CalDAV/CardDAV server -- **Syncthing** - Decentralized file sync -- **Heyform** - Form builder and surveys -- **Dawarich** - Location tracking -- **Palmr** - File sharing service -- **Obsidian LiveSync** - CouchDB backend for note synchronization +### Media (`media`) +| Tag | Service | Notes | +|---|---|---| +| `audiobookshelf` | Audiobookshelf | Pocket ID set up in the app's settings. Podcasts are not used. | +| `kavita` | Kavita | Ebooks at `books.`. Also handles OPDS and KOReader sync (replaced Calibre-Web, dir2opds and kosync). Pocket ID set up in the app's settings. | +| `karakeep`, `hoarder` | Karakeep | Files are still named `hoarder*`. Web, headless Chrome and Meilisearch. Pocket ID through `hoarder-env.j2`. | +| `manyfold` | Manyfold | Pocket ID through env vars. | -### Communication (`communication/`) -- **GoToSocial** - Lightweight ActivityPub server -- **Postiz** - Social media management +### Productivity (`productivity`) +| Tag | Service | Notes | +|---|---|---| +| `paperlessngx` | Paperless-ngx | Redis, Gotenberg, Tika. Pocket ID through django-allauth (`paperlessngx.env.j2`). Daily S3 backup of the data volume. | +| `baikal` | Baikal | CalDAV/CardDAV. Caddy redirects the `.well-known` paths. | +| `syncthing` | Syncthing | Ports on the Tailscale IP only. | +| `heyform` | HeyForm | MongoDB (Percona) and KeyDB. On `forms.nerder.land`. | +| `dawarich` | Dawarich | App, Sidekiq, PostGIS, Redis. Based on the upstream production compose file. | +| `obsidian-livesync` | CouchDB for Obsidian LiveSync | Port 5984 on the Tailscale IP only. Has a restart handler. | +| `gokapi` | Gokapi | Pocket ID login. `config.json` is only written on a fresh install, because Gokapi rewrites it on upgrade. To change it, delete `/opt/stacks/gokapi/config/config.json` and redeploy. | -### Monitoring (`monitoring/`) -- **Change Detection** - Website monitoring -- **Apprise API** - Unified notifications -- **Gotify** - Self-hosted push notification service +### Communication (`communication`) +| Tag | Service | Notes | +|---|---|---| +| `gotosocial` | GoToSocial | Pocket ID through env vars. Account domain is the apex; Caddy serves WebFinger there. Daily S3 backup of its volume. | +| `listmonk` | Listmonk | Postgres 17. On `listmonk.nerder.land`. | +| `starpunk` | StarPunk | Micropub server for the main site (own image from Gitea). Logs in through Gondulf. | -## Deployment Patterns +### Monitoring (`monitoring`) +| Tag | Service | Notes | +|---|---|---| +| `changedetection` | Changedetection | With a sockpuppetbrowser for JavaScript pages. | +| `appriseapi` | Apprise API | Port 8000 on the Tailscale IP only. Other containers reach it at `http://apprise:8000`. | +| `gotify` | Gotify + iGotify assistant | Push notifications. Dockcheck sends its update notices here. | +| `cronmaster` | Cronmaster | Cron UI, port 40123 on the Tailscale IP only. | -### Standardized Service Deployment -Each service follows a consistent pattern: -1. Creates `/opt/stacks/[service-name]` directory structure -2. Generates Docker Compose file from Jinja2 template -3. Deploys using `community.docker.docker_compose_v2` -4. Configures environment variables from vault secrets +## Networking -### Template System -- **Compose Templates**: `.j2` files in `templates/` for dynamic configuration -- **Environment Templates**: Separate `.env.j2` files for services requiring environment variables -- **Variable Substitution**: Uses centralized variable hierarchy from group_vars structure -- **Domain Management**: Centralized domain and subdomain configuration -- **Network Configuration**: Standardized Docker network and IP address management +### Shared network (`lava`) +Every stack joins the external network `lava` (`172.20.0.0/16`). Docker only +hands out dynamic addresses from `172.20.128.0/17`, so `172.20.0.x` is free +for fixed IPs: Caddy (.5) and Gitea (.6). Both need fixed IPs because the +ufw-docker rules point at container IPs. The role creates the network if it +is missing and warns if it lacks the reserved range; `migrate-lava-network.yml` +fixes an old network. -## Shell Environment Setup -The role also configures the shell environment: -- **Zsh Installation**: Sets zsh as default shell -- **Atuin**: Command history sync and search -- **Bat**: Enhanced `cat` command with syntax highlighting +### Hairpinning +Containers can't reach the server's own public IP, so any app that calls +another of our domains (usually Pocket ID for OIDC) maps it to Caddy: + +```yaml +extra_hosts: + - "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}" +``` + +### Firewall +`ufw-docker.sh` makes UFW apply to published container ports. Only Caddy +(80/tcp, 443/tcp, 443/udp) and Gitea (22/tcp, published as 222) are allowed +in from the internet; the Caddy and Gitea task files add those rules. + +## Updates + +Containers labelled `mag37.dockcheck.update: true` are updated automatically +each morning by dockcheck (set up in the cron role). Containers without the +label (Gondulf, StarPunk, Listmonk, HeyForm, Dockge and most databases) only +get a Gotify notice. Dawarich's database and Redis are skipped entirely. +Gondulf and StarPunk use pinned version tags. + +## Adding a service + +1. `templates/-compose.yml.j2` (plus `-env.j2` if needed). Join the + external `{{ docker.network_name }}` network and add `extra_hosts` for any + of our domains the app calls. +2. `tasks//.yml`: make `/opt/stacks/`, template + the files, deploy with `community.docker.docker_compose_v2`. Use + `diff: false` or `no_log: true` on tasks that write secrets. +3. Import it in `tasks//main.yml` with a tag named after the service. +4. Subdomain in `group_vars/all/domains.yml`, route in `templates/Caddyfile.j2`, + A record in `dns.yml`. +5. Secrets under `vault_` with `bin/vault-run`. + +## Files -## File Organization ``` roles/docker/ ├── tasks/ -│ ├── main.yml # Orchestrates all deployments -│ ├── shell.yml # Shell environment setup -│ ├── infrastructure/ -│ │ ├── main.yml # Infrastructure category orchestrator -│ │ ├── caddy.yml # Reverse proxy -│ │ └── pocketid.yml # Authentication -│ ├── development/ -│ │ ├── main.yml # Development category orchestrator -│ │ ├── gitea.yml # Git hosting -│ │ └── bytestash.yml # Code snippets -│ ├── media/ # Media services (7 services) -│ ├── productivity/ # Productivity services (7 services) -│ ├── communication/ # Communication services (2 services) -│ └── monitoring/ # Monitoring services (3 services) -├── templates/ -│ ├── [service]-compose.yml.j2 # Docker Compose templates (all templated) -│ ├── [service]-env.j2 # Environment variable templates -│ └── [service]-*.j2 # Service-specific templates -├── files/ -│ ├── Caddyfile # Caddy configuration -│ ├── ufw-docker.sh # Firewall integration script -│ ├── client # Matrix well-known client file -│ └── server # Matrix well-known server file -└── handlers/ - └── main.yml # Service restart handlers +│ ├── main.yml # System setup, network, category imports +│ ├── shell.yml # zsh, Atuin, bat +│ ├── infrastructure/ # caddy, pocketid, gondulf (+ dockge in main.yml) +│ ├── development/ # gitea, bytestash +│ ├── media/ # audiobookshelf, kavita, hoarder (Karakeep), manyfold +│ ├── productivity/ # paperlessngx, baikal, syncthing, heyform, dawarich, obsidian-livesync, gokapi +│ ├── communication/ # gotosocial, listmonk, starpunk +│ └── monitoring/ # changedetection, appriseapi, gotify, cronmaster +├── templates/ # -compose.yml.j2, env and config templates, Caddyfile.j2 +├── files/ufw-docker.sh # UFW integration for Docker +└── handlers/main.yml # restart caddy, restart obsidian-livesync ``` -## Usage - -### Deploy All Services -```bash -ansible-playbook site.yml -i hosts.yml --tags docker -``` - -### Deploy by Service Category -```bash -# Deploy entire service categories -ansible-playbook site.yml -i hosts.yml --tags infrastructure -ansible-playbook site.yml -i hosts.yml --tags development -ansible-playbook site.yml -i hosts.yml --tags media -ansible-playbook site.yml -i hosts.yml --tags productivity -ansible-playbook site.yml -i hosts.yml --tags communication -ansible-playbook site.yml -i hosts.yml --tags monitoring - -# Deploy multiple categories -ansible-playbook site.yml -i hosts.yml --tags infrastructure,monitoring -``` - -### Deploy Individual Services -```bash -# Deploy specific services -ansible-playbook site.yml -i hosts.yml --tags pocketid -ansible-playbook site.yml -i hosts.yml --tags gitea,bytestash -ansible-playbook site.yml -i hosts.yml --tags paperlessngx -``` - -## Service-Specific Notes - -### MMDL (Task Management) -- **URL**: https://tasks.thesatelliteoflove.com -- **Initial Setup**: Visit `/install` endpoint first to run database migrations -- **Authentication**: Integrates with the Pocket ID OIDC provider -- **Database**: Uses MySQL 8.0 with automatic schema migration -- **Features**: CalDAV integration, multiple account support, task management - -## Dependencies - -### System Requirements -- Docker CE installed and running -- UFW firewall configured -- DNS records pointing to the server -- Valid SSL certificates (handled automatically by Caddy) - -### External Services -- **DNS**: Requires subdomains configured for each service -- **Email**: Gitea uses Resend for notifications -- **Storage**: All services persist data to `/opt/stacks/[service]/` - -## Configuration - -### Variable Structure -The role uses a centralized variable hierarchy in `group_vars/all/`: - -- **domains.yml**: Domain and subdomain mappings for all services -- **infrastructure.yml**: Network configuration, Docker settings, and system parameters -- **vault.yml**: Encrypted secrets including API keys, passwords, and OAuth credentials -- **services.yml**: Service-specific configuration and feature flags - -### Required Variables (in vault.yml) -- Authentication credentials for various services (vault_*) -- API keys for external integrations -- OAuth client secrets for SSO integration -- Database passwords and connection strings -- SMTP credentials for notifications - -### Network Configuration -Services expect to be accessible via subdomains of configured domains: -- `id.thesatelliteoflove.com` - Pocket ID -- `git.thesatelliteoflove.com` - Gitea -- `books.thesatelliteoflove.com` - Kavita -- `tasks.thesatelliteoflove.com` - MMDL -- (and many more...) - -## Monitoring & Management - -### Operational Features -- Automatic container restart policies -- Health checks for database services -- Centralized logging and monitoring -- Backup-ready data structure in `/opt/stacks/` - -## Security Considerations - -### Network Security -- UFW-Docker integration for proper firewall rules -- Services isolated to appropriate network segments -- Restricted access for sensitive tools (Stirling PDF) - -### Authentication -- Centralized SSO through Pocket ID for most services -- OAuth integration where supported -- Secure secret management through Ansible vault - -### Container Security -- Non-root container execution (UID/GID 1000:1000) -- Security options: `no-new-privileges: true` -- Capability dropping and minimal permissions - ## Troubleshooting -### Common Issues -- **Database Connection**: Ensure MySQL containers use proper authentication plugins -- **OAuth Discovery**: Check issuer URLs don't have trailing slashes -- **Migration Failures**: Visit service `/install` endpoints for database setup -- **Network Issues**: Verify containers are on the same Docker network \ No newline at end of file +- **OIDC login fails with a connection error:** the container is missing the + `extra_hosts` entry for `id.thesatelliteoflove.com`. +- **OIDC discovery fails:** check the issuer URL has no trailing slash. +- **Caddy fails with "Address already in use":** another container took + 172.20.0.5; the network lacks the reserved range. Run + `migrate-lava-network.yml`. +- **Gitea SSH stops working after a recreate:** Gitea lost its fixed IP; + redeploy with `--tags gitea` to restore it and the firewall rule. diff --git a/todo.md b/todo.md index fd72ec9..3b6ef7a 100644 --- a/todo.md +++ b/todo.md @@ -1,141 +1,58 @@ # Infrastructure Improvements TODO -## High Priority (Quick Wins) +Current state (2026-10-31): 24 services in 6 categories, all compose files +templated, Pocket ID as SSO (Authentik retired), multi-agent guard rails +(`bin/worktree`, `bin/deploy`, `bin/vault-run`, `deploy-guard.yml`). -### 1. Split the massive docker role ✅ COMPLETED -- **Current Issue**: `roles/docker/tasks/main.yml` has 20+ services in one file (176 lines) -- **Solution**: Break into logical service groups: - ``` - roles/docker/tasks/ - ├── main.yml (orchestrator) - ├── infrastructure/ (caddy, pocketid, dockge) - ├── development/ (gitea, codeserver, bytestash) - ├── media/ (audiobookshelf, kavita, ghost, pinchflat, pinry, hoarder, manyfold) - ├── productivity/ (paperless, baikal, syncthing, mmdl, heyform, dawarich, palmr, obsidian-livesync) - ├── communication/ (gotosocial, postiz) - └── monitoring/ (changedetection, appriseapi, gotify) - ``` -- **COMPLETED**: All services organized into logical categories with category-level tags +## Done +- Split the docker role into categories with category and service tags. +- Central variables in `group_vars/all/` (`domains.yml`, `infrastructure.yml`, + `vault.yml`). +- Converted every static compose file to a template. +- Removed unused services, most recently Authentik, Calibre-Web, dir2opds, + kosync, Sneaky Klaus and Audiobookshelf podcasts. +- Reserved address range on the `lava` network so fixed IPs can't be taken. +- Automatic container updates with dockcheck for labelled containers. -### 2. Standardize variable management ✅ COMPLETED -- **Current Issue**: Secrets in single encrypted file, no clear variable hierarchy -- **Solution**: Create proper variable structure: - ``` - group_vars/ - ├── all/ - │ ├── domains.yml (domain and subdomain mappings) - │ ├── infrastructure.yml (network config, Docker settings) - │ ├── vault.yml (encrypted secrets with vault_ prefix) - │ └── services.yml (service configuration flags) - ``` -- **COMPLETED**: Implemented comprehensive variable hierarchy, updated all templates to use centralized variables, fixed service tag isolation +## Open -### 3. Template consolidation ✅ PARTIALLY COMPLETED -- **Current Issue**: Many compose templates repeat patterns, some services used static files -- **Solution**: Create reusable template includes with standard service template structure -- **COMPLETED**: Converted all static compose files (caddy, dockge, hoarder) to Jinja2 templates -- **REMAINING**: Create reusable template patterns for common configurations +### Reusable template patterns +Many compose templates repeat the same network, restart and label blocks. +Pull the shared parts into includes or macros. -## Security & Reliability +### Health checks +Only Pocket ID, Gondulf, Dawarich, Listmonk and Syncthing define health checks +in their templates. Add them to the rest where the image supports it. -### 4. Add health checks -- **Issue**: Most services lack proper healthcheck configurations in compose templates -- **Solution**: Implement comprehensive health monitoring with standardized healthcheck patterns +### Backups +Only Paperless-ngx and GoToSocial back up (daily to S3 with +`offen/docker-volume-backup`, 7-day retention). Still to do: +- Database dumps for the Postgres/MongoDB/CouchDB services. +- Volume backups for the file-based services (Kavita, Audiobookshelf, Gitea, + Pocket ID, …). +- A tested restore. -### 5. Implement backup strategy -- **Issue**: No automated backups for 25 services and their data -- **Solution**: Add backup role with: - - Database dumps for PostgreSQL services - - Volume backups for file-based services - - Rotation policies - - Restoration testing +### Network segmentation +Every service shares the `lava` network. Consider separate frontend, backend +and database networks. -### 6. Network segmentation -- **Issue**: All services share one Docker network -- **Solution**: Separate into: - - `frontend` (Public-facing services) - - `backend` (Internal services only) - - `database` (Database access only) +### Security hardening +- Add `no-new-privileges` and dropped capabilities where images allow it + (only Manyfold has them now). Cronmaster runs as root. +- fail2ban (or Caddy rate limits) in front of login pages. -### 7. Security hardening -- Remove unnecessary `user: root` from services -- Add security contexts to all containers -- Implement least-privilege access patterns -- Add fail2ban for authentication services +### CI with Gitea Actions +Syntax check and `--check` runs on push, using the existing Gitea runner. -## Automation Opportunities +### Configuration drift detection +Catch manual changes on the server, e.g. a scheduled `bin/deploy --check`. -### 8. CI/CD with Gitea Actions -- Leverage self-hosted Gitea for: - - Ansible syntax validation - - Service configuration testing - - Automated deployment triggers - - Rollback capabilities +### Service start order +Apps that use Pocket ID don't wait for it to be up. -### 9. Configuration drift detection -- Add validation tasks to catch manual changes -- Implement configuration validation with proper assertions - -### 10. Service dependency management -- **Issue**: Some services depend on Pocket ID SSO but no startup ordering -- **Solution**: Implement dependency checking and startup ordering - -### 11. Ansible best practices -- Replace deprecated `apt_key` with proper patterns -- Use `ansible.builtin` FQCN consistently -- Add `check_mode` support -- Implement proper idempotency checks - -### 12. Documentation automation -- Auto-generate service inventory -- Create service documentation templates -- Implement automated documentation updates - -## Implementation Roadmap - -### Week 1: Foundation -- [x] Document improvements in todo.md -- [x] Reorganize docker role structure -- [x] Convert static compose files to templates -- [x] Remove unused services (beaver, grist, stirlingpdf, tasksmd, redlib) -- [x] Clean up templates and files directories -- [x] Implement variable hierarchy -- [ ] Create reusable template patterns - -### Week 2: Security & Monitoring -- [ ] Add health checks -- [ ] Implement backup strategy -- [ ] Security hardening - -### Week 3: Automation -- [ ] CI/CD pipeline setup -- [ ] Configuration validation -- [ ] Documentation automation - -### Week 4: Advanced Features -- [ ] Network segmentation -- [ ] Dependency management -- [ ] Monitoring dashboard - -## Completed Work Summary - -### ✅ Major Accomplishments -- **Docker Role Reorganization**: Split monolithic 176-line main.yml into 6 logical service categories -- **Template Standardization**: Converted all static compose files to Jinja2 templates -- **Service Cleanup**: Removed 5 unused/broken services (beaver, grist, stirlingpdf, tasksmd, redlib) -- **Category-Based Deployment**: Can now deploy services by category using tags (infrastructure, media, etc.) -- **Variable Management**: Implemented comprehensive centralized variable hierarchy with proper secret organization -- **Service Tag Isolation**: Fixed service tags to deploy individual services only (not entire categories) -- **Documentation Updates**: Updated all README files and CLAUDE.md to reflect new architecture - -### 📊 Current Stats -- **25 active services** organized into 6 categories -- **100% templated** compose files (no static files) -- **6 service directories** for logical organization -- **Clean file structure** with only essential static files - -## Notes -- Current architecture is solid and much better organized for long-term maintainability -- Focus on high-impact, low-effort improvements first -- Leverage existing infrastructure (Gitea, Pocket ID) for automation -- Template-driven approach enables future dynamic configuration \ No newline at end of file +### Ansible clean-up +- Replace the deprecated `apt_key` task for Docker's repo (it also fetches the + Ubuntu key for a Debian repo). +- Use `ansible.builtin.` names everywhere. +- The Dockge compose file is templated twice (`tasks/main.yml` and + `tasks/infrastructure/main.yml`).