phil/ansible
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bring the docs up to date with the running stack

Browse Source 
The READMEs, CLAUDE.md and todo.md still described Authentik, services
removed long ago (Ghost, MMDL, Postiz, Pinchflat...) and two servers. Now
they list the 24 services actually deployed (checked against docker ps),
explain that netcup and docker-01 are one machine, document Gondulf,
StarPunk, Listmonk, Gokapi and dockcheck, and use bin/deploy throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Phil
2026-10-31 11:30:50 -06:00
co-authored by Claude Opus 5.5
parent 0282debf07
commit 0c14ee5b1d
8 changed files with 426 additions and 639 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.local.md
+1 -1
View File
@@ -4,4 +4,4 @@
- Whenever I talk about cron jobs I am referring to cron jobs on the remote servers managed by ansible, never the local machine
- never use secrets.enc
- all secrets go in vault.yml, never secrets.enc, never some random file you want to create, only ever vault.yml. you decrypt vault.yml with vault_pass
- Never use ansible-vault edit. always decrypt, make the changes, then encrypt
- Never use ansible-vault edit. Edit vault.yml only with bin/vault-run <command> (it decrypts, runs your edit, checks and re-encrypts)
CLAUDE.md
+89 -118
View File
@@ -4,9 +4,13 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Overview
This is a personal infrastructure Ansible playbook that automates deployment and management of 21 self-hosted Docker services across two domains (`thesatelliteoflove.com` and `nerder.land`). The setup uses Tailscale VPN for secure networking and Caddy for reverse proxy with automated HTTPS.
Personal infrastructure as Ansible: 24 self-hosted Docker services on one
server, across `thesatelliteoflove.com` and `nerder.land`. Caddy is the
reverse proxy (automatic HTTPS), Pocket ID is the passkey SSO, and Tailscale
is how Ansible reaches the server. See `README.md` for the service list.
**Important**: Always review `DEPLOYMENT_LEARNINGS.md` when working on this repository for lessons learned and troubleshooting guidance.
If a local `DEPLOYMENT_LEARNINGS.md` exists (it's gitignored), read it for
past lessons.
## Working alongside other agents (read first)
@@ -41,156 +45,123 @@ other. Follow these steps:
## Common Commands
### Initial Setup
```bash
# Install Ansible dependencies
ansible-galaxy install -r requirements.yml
bin/worktree <branch> # start a task (see above)
bin/deploy --tags <service> # deploy one service
bin/deploy --tags media,productivity # deploy categories
bin/deploy --tags services # every service, no system setup
bin/deploy --tags common,cron # base setup and cron jobs only
bin/deploy --check --diff --tags <tag> # dry run, allowed from any checkout
PLAYBOOK=dns.yml bin/deploy # Route53 records
bin/vault-run <command> # edit secrets ($VAULT_PLAIN)
# Bootstrap new server (creates user, installs Tailscale, security hardening)
ansible-playbook bootstrap.yml -i hosts.yml
# Deploy all Docker services
ansible-playbook site.yml -i hosts.yml
# Update DNS records in AWS Route53
ansible-playbook dns.yml -i hosts.yml
# First-time setup of a new server (public IP, before Tailscale)
.venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass
```
### Service Management
```bash
# Deploy specific services using tags (now properly isolated)
ansible-playbook site.yml -i hosts.yml --tags caddy --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags pocketid --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags gitea --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags docker --vault-password-file vault_pass # entire docker role
ansible-playbook site.yml -i hosts.yml --tags services --vault-password-file vault_pass # all services only
# Deploy services by category
ansible-playbook site.yml -i hosts.yml --tags infrastructure --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags media,productivity --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags development,monitoring --vault-password-file vault_pass
# Deploy only infrastructure components
ansible-playbook site.yml -i hosts.yml --tags common,cron --vault-password-file vault_pass
```
The venv (`.venv/`), `vault_pass` and `group_vars/all/vault.yml` live in the
main checkout; the `bin/` helpers find them from any worktree.
## Architecture
### Host Configuration
- **Bootstrap Host** (`netcup`): 152.53.36.98 - Initial server setup target
- **Docker Host** (`docker-01`): 100.70.169.99 - Main service deployment via Tailscale
### Host
One server, `cogitator-01` (netcup VPS), listed twice in `hosts.yml`:
- `netcup` (group `bootstrap`): public IP 152.53.36.98, used only by `bootstrap.yml`.
- `docker-01` (group `docker`): Tailscale IP 100.70.169.99, used by `site.yml`.
### Role Structure
- **bootstrap**: Initial server hardening, user creation, Tailscale VPN setup
- **common**: Basic system configuration, UFW firewall management
- **docker**: Comprehensive service deployment (21 containerized applications, organized by category)
- **cron**: Scheduled task management (currently Warhammer RSS feed generation)
DNS A records point at the public IP; Caddy on the same machine routes by
subdomain.
### Docker Role Organization (Reorganized into Logical Categories)
The docker role is now organized into logical service groups under `roles/docker/tasks/`:
### Roles (`site.yml` runs common, docker, cron)
- **bootstrap**: user, SSH key, passwordless sudo, UFW, then `artis3n.tailscale`.
- **common**: `aptitude`, `jq`, UFW on with default deny.
- **docker**: Docker CE, shell setup (zsh, Atuin, bat), ufw-docker, the `lava`
network, and every service.
- **cron**: dockcheck (08:00, auto-updates labelled containers, Gotify notices)
and the Warhammer RSS feed (09:10).
- **infrastructure/**: Core platform components
- Caddy (reverse proxy), Pocket ID (passkey SSO), Dockge (container management)
- **development/**: Development and collaboration tools
- Gitea, ByteStash
- **media/**: Content creation and consumption
- Audiobookshelf, Kavita, Karakeep (formerly Hoarder), Manyfold
- **productivity/**: Personal organization and document management
- Paperless-NGX, Baikal (CalDAV/CardDAV), Syncthing, Heyform, Dawarich, Obsidian LiveSync
- **communication/**: Social media
- GoToSocial (Fediverse)
- **monitoring/**: System monitoring and alerts
- Changedetection, AppriseAPI, Gotify, Cronmaster
### Docker role categories (`roles/docker/tasks/<category>/`)
- **infrastructure/**: Caddy, Pocket ID, Gondulf (IndieAuth), Dockge
- **development/**: Gitea (+ act_runner), ByteStash
- **media/**: Audiobookshelf, Kavita, Karakeep (files named `hoarder`), Manyfold
- **productivity/**: Paperless-ngx, Baikal, Syncthing, HeyForm, Dawarich, Obsidian LiveSync, Gokapi
- **communication/**: GoToSocial, Listmonk, StarPunk (Micropub)
- **monitoring/**: Changedetection, Apprise API, Gotify (+ iGotify), Cronmaster
### Variable Management
**Critical**: This infrastructure uses a centralized variable hierarchy in `group_vars/all/`:
Each category has a tag (`infrastructure`, `media`, …) and each service its
own tag; `services` covers all of them. Karakeep has both `karakeep` and
`hoarder` tags.
- **domains.yml**: Domain and subdomain mappings (use `{{ subdomains.service }}`)
- **infrastructure.yml**: Network configuration, Docker settings (`{{ docker.network_name }}`, `{{ docker.hairpin_ip }}`)
- **vault.yml**: Encrypted secrets with `vault_` prefix
- **services.yml**: Service-specific configuration and feature flags
### Variables (`group_vars/all/`)
- **domains.yml**: `primary_domain`, `secondary_domain`, `subdomains.<service>`, email domains.
- **infrastructure.yml**: `docker.*` (network name, subnet, fixed IPs), `smtp.*` (Resend), `network.*` (public and Tailscale IPs).
- **vault.yml**: encrypted secrets, one `vault_<service>` dict per service.
**Important**: All templates use variables instead of hardcoded values. Never hardcode domains, IPs, or secrets.
Never hardcode domains, IPs or secrets in templates; use these variables.
### Data Structure
- All service data stored in `/opt/stacks/[service-name]/` on docker host
- Docker Compose files generated from Jinja2 templates in `roles/docker/templates/`
- Environment files templated for services requiring configuration
- All configurations use centralized variables for consistency
### Data
- Each service lives in `/opt/stacks/<service>/` on the server (Dockge in `/opt/dockge/`).
- Compose files come from `roles/docker/templates/<service>-compose.yml.j2`; env and config files from `<service>-env.j2` / `.env.j2` / `-config.json.j2`.
- Retired services' final backups are in `/opt/stacks/_retired/`.
## Key Implementation Details
### Template-Driven Configuration
The docker role uses Jinja2 templates exclusively for all services. When modifying services:
- Update templates in `roles/docker/templates/[service]-compose.yml.j2`
- Environment files use `.env.j2` templates where needed
- Task files organized by category in `roles/docker/tasks/[category]/[service].yml`
- All services now use templated configurations (no static compose files)
### Adding or changing a service
- Template: `roles/docker/templates/<service>-compose.yml.j2`.
- Tasks: `roles/docker/tasks/<category>/<service>.yml`, imported with a tag in that category's `main.yml`.
- Public access: subdomain in `domains.yml`, route in `Caddyfile.j2`, A record in `dns.yml`.
- Use `diff: false` / `no_log: true` on tasks that write secrets.
- Add the label `mag37.dockcheck.update: true` if dockcheck may update it unattended.
### DNS Management
The `dns.yml` playbook manages AWS Route53 records for both domains. All subdomains point to the netcup server (152.53.36.98), with Caddy handling internal routing to the docker host via Tailscale.
### DNS
`dns.yml` manages Route53 A records for both domains with a Route53-only IAM
user (`vault_aws_dns`). It also deletes the records listed under
`removed_records`; add a retired service's subdomain there.
### Security Architecture
- Tailscale provides secure networking between management and service hosts
- Services are network-isolated using Docker
- Caddy handles SSL termination with automatic Let's Encrypt certificates
- UFW firewall managed through Docker integration script
### Security
- Caddy terminates TLS with Let's Encrypt.
- `ufw-docker` makes UFW govern container ports. Only Caddy (80/443) and
Gitea SSH (host port 222) are allowed in from the internet.
- Admin UIs (Syncthing, Cronmaster, Apprise, CouchDB) bind to the Tailscale IP only.
### Service Dependencies
Most services log in through Pocket ID (OIDC, passkeys) at `{{ subdomains.pocketid }}`. When deploying new services, consider:
- Whether SSO integration is needed
- Caddy routing configuration for subdomain access
- Network connectivity requirements within Docker stack
- Hairpinning fixes for internal service-to-service communication
### SSO (Pocket ID)
Pocket ID (`id.thesatelliteoflove.com`) replaced Authentik on 2026-10-31.
Apps that log in with it: Gitea, ByteStash, Audiobookshelf, Kavita, Karakeep,
Manyfold, Paperless-ngx, Gokapi, GoToSocial. Some read the client from env
vars; Gitea, Audiobookshelf and Kavita store it in their own database (set in
their admin UI). OIDC clients are created in Pocket ID's UI and their ID and
secret go in the vault as `vault_<service>.pocketid`. StarPunk logs in with
IndieAuth through Gondulf instead.
### Shared Docker Network (`lava`)
Created by the docker role as `{{ docker.subnet }}` with `ip_range: {{ docker.ip_range }}`. Docker only hands out dynamic addresses from that range, so `172.20.0.x` is reserved for fixed IPs: Caddy (`docker.hairpin_ip`, .5) and Gitea (`docker.gitea_ip`, .6). Both need fixed IPs because ufw-docker firewall rules point at container IPs. A network without the range (the 2024 original) is migrated once with `migrate-lava-network.yml`.
### Hairpinning Resolution
Services inside Docker containers cannot reach external domains that resolve to the same server. Fix by adding `extra_hosts` mappings:
### Hairpinning
Containers can't reach domains that resolve to this server's public IP. Map
any of our domains a container calls to Caddy with `extra_hosts`:
```yaml
extra_hosts:
- "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}"
- "{{ subdomains.cal }}:{{ docker.hairpin_ip }}"
```
Common domains requiring hairpinning fixes:
- `{{ subdomains.pocketid }}` (Pocket ID SSO)
- `{{ subdomains.cal }}` (Baikal CalDAV)
- Any service domain the container needs to communicate with
Most often needed for `subdomains.pocketid` (OIDC); also `subdomains.git`
(Gitea itself), `subdomains.gondulf` and `primary_domain` (StarPunk).
**Note**: Use variables instead of hardcoded values for maintainability.
### Template pattern
```yaml
networks:
default:
external: true
name: "{{ docker.network_name }}"
```
### Service-Specific Reference Configurations
- **Dawarich**: Based on production compose file at https://github.com/Freika/dawarich/blob/master/docker/docker-compose.production.yml
### Service-Specific Notes
- **Dawarich**: based on https://github.com/Freika/dawarich/blob/master/docker/docker-compose.production.yml
- **Gokapi**: `config.json` is only written on a fresh install (`force: false`); delete it on the server to apply template changes.
- **Caddy**: Caddyfile changes restart it via a handler; the stack uses `recreate: never`.
## Service Memories
- karakeep (formerly called hoarder) is deployed with both 'hoarder' and 'karakeep' tags for backward compatibility
- whenever i ask you what containers need updates, run dockcheck and return a list of containers needing updates
- when i ask for the status container updates i want you to run dockcheck on the docker host https://github.com/mag37/dockcheck?ref=selfh.st
## Variable Management Implementation Notes
**Major Infrastructure Update**: Variable management system was implemented to replace all hardcoded values with centralized variables.
### Key Changes Made:
- Created comprehensive `group_vars/all/` structure
- Updated all Docker Compose templates to use variables
- Fixed service tag isolation (individual service tags now deploy only that service)
- Standardized domain and network configuration
- Organized secrets by service with consistent `vault_` prefix
### Service Tag Fix:
**Critical**: Service tags are properly isolated. Individual service tags deploy only that service. The `services` tag runs all service tasks across all categories.
### Template Pattern:
All templates now follow this pattern:
```yaml
# Use variables, not hardcoded values
networks:
default:
external: true
name: "{{ docker.network_name }}"
extra_hosts:
- "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}"
```
README.md
+117 -120
View File
@@ -1,157 +1,154 @@
# Personal Infrastructure Ansible Playbook
This Ansible playbook automates the setup and management of a personal self-hosted infrastructure running Docker containers for various services.
Ansible that builds and runs a personal self-hosted stack: 24 Docker services
on one server, served on `thesatelliteoflove.com` and `nerder.land`, with
Caddy for HTTPS and Pocket ID (passkeys) for single sign-on.
## Overview
## The server
The playbook manages two main environments:
- **Bootstrap server** (`netcup`): Initial server setup with Tailscale VPN
- **Docker server** (`docker-01`): Main application server running containerized services
There is one server (hostname `cogitator-01`, a netcup VPS). The inventory
lists it twice, under two names:
## Services Deployed
| Inventory name | Address | Used by |
|---|---|---|
| `netcup` (group `bootstrap`) | 152.53.36.98 (public IP) | `bootstrap.yml`, for first-time setup before Tailscale is up |
| `docker-01` (group `docker`) | 100.70.169.99 (Tailscale IP) | `site.yml`, for everything after that |
The Docker role deploys and manages 21 self-hosted services organized into logical categories:
All public DNS records point at the public IP. Caddy on the same machine
takes ports 80/443 and passes each subdomain to the right container.
### Infrastructure
- **Caddy** (Reverse proxy with automatic HTTPS)
- **Pocket ID** (passkey SSO/Identity Provider)
- **Dockge** (Container management)
## Services
### Development
- **Gitea** (Git repository hosting)
- **ByteStash** (Code snippet management)
| Category | Service | Address | Login |
|---|---|---|---|
| Infrastructure | Caddy (reverse proxy, static sites) | — | — |
| | Pocket ID (passkey OIDC provider) | id.thesatelliteoflove.com | passkeys |
| | Gondulf (IndieAuth for the main site) | gondulf.thesatelliteoflove.com | email code |
| | Dockge (compose stack manager) | port 5001, no public route | local |
| Development | Gitea + Actions runner | git.thesatelliteoflove.com (SSH on port 222) | Pocket ID |
| | ByteStash (code snippets) | snippets.thesatelliteoflove.com | Pocket ID |
| Media | Audiobookshelf | audio.thesatelliteoflove.com | Pocket ID |
| | Kavita (ebooks, OPDS, KOReader sync) | books.thesatelliteoflove.com | Pocket ID |
| | Karakeep (bookmarks, formerly Hoarder) | bookmarks.thesatelliteoflove.com | Pocket ID |
| | Manyfold (3D models) | models.thesatelliteoflove.com | Pocket ID |
| Productivity | Paperless-ngx (documents) | paper.thesatelliteoflove.com | Pocket ID |
| | Baikal (CalDAV/CardDAV) | cal.thesatelliteoflove.com | local |
| | Syncthing | Tailscale only, port 8384 | local |
| | HeyForm (forms) | forms.nerder.land | local |
| | Dawarich (location history) | loclog.thesatelliteoflove.com | local |
| | Obsidian LiveSync (CouchDB) | Tailscale only, port 5984 | local |
| | Gokapi (file sharing) | gokapi.thesatelliteoflove.com | Pocket ID |
| Communication | GoToSocial (fediverse) | social.thesatelliteoflove.com | Pocket ID |
| | Listmonk (mailing lists) | listmonk.nerder.land | local |
| | StarPunk (Micropub for the main site) | starpunk.thesatelliteoflove.com | IndieAuth |
| Monitoring | Changedetection | watcher.thesatelliteoflove.com | local |
| | Apprise API (notifications) | Tailscale only, port 8000 | — |
| | Gotify + iGotify assistant | gotify.thesatelliteoflove.com, gotify-assistant.thesatelliteoflove.com | local |
| | Cronmaster (cron UI) | Tailscale only, port 40123 | local |
### Media
- **Audiobookshelf** (Audiobook server)
- **Kavita** (E-book server)
- **Karakeep** (Bookmark manager, formerly Hoarder)
- **Manyfold** (3D model organizer)
"Tailscale only" means the port is bound to the Tailscale IP and has no public
address.
### Productivity
- **Paperless-NGX** (Document management)
- **Baikal** (CalDAV/CardDAV server)
- **Syncthing** (File synchronization)
- **HeyForm** (Form builder)
- **Dawarich** (Location tracking)
- **Obsidian LiveSync** (Note synchronization)
Caddy also serves static sites from `/opt/stacks/caddy/site/` (the files are
not managed by Ansible): the apex `thesatelliteoflove.com` (plus WebFinger for
the fediverse and Tailscale login), `pdg.thesatelliteoflove.com`,
`nerder.land` and `repair.nerder.land`. `repaircafe.nerder.land` redirects to
the Listmonk signup form.
### Communication
- **GoToSocial** (Fediverse/Mastodon)
### Monitoring
- **Changedetection** (Website change monitoring)
- **AppriseAPI** (Notification service)
- **Gotify** (Push notifications)
- **Cronmaster** (Scheduled container tasks)
## Structure
## Layout
```
├── site.yml # Main playbook
├── bootstrap.yml # Server bootstrap playbook
├── dns.yml # AWS Route53 DNS management
├── hosts.yml # Inventory file
├── requirements.yml # External role dependencies
├── site.yml # Main playbook: common, docker and cron roles
├── bootstrap.yml # First-time server setup (user, SSH key, UFW, Tailscale)
├── dns.yml # Route53 A records (adds current ones, removes retired ones)
├── deploy-guard.yml # Stops real runs from anything but a clean origin/main
├── migrate-lava-network.yml # One-off: recreate the shared docker network
├── hosts.yml # Inventory
├── requirements.yml # Galaxy roles (artis3n.tailscale)
├── bin/
│ ├── worktree # Start a task in its own git worktree
│ ├── deploy # Run a playbook while holding the deploy lock
│ └── vault-run # Edit vault.yml safely
├── group_vars/all/
│ ├── domains.yml # Domains and subdomain map
│ ├── infrastructure.yml # Docker network, IPs, SMTP, paths
│ └── vault.yml # Encrypted secrets (gitignored)
└── roles/
├── bootstrap/ # Initial server setup
├── common/ # Common server configuration
├── cron/ # Scheduled tasks
└── docker/ # Docker services deployment
├── bootstrap/ # First-time setup
├── common/ # Base packages and UFW
├── docker/ # Docker, shell setup and every service
└── cron/ # Host cron jobs (dockcheck, Warhammer feed)
```
## Roles Documentation
Each role has detailed documentation in its respective directory:
### [Bootstrap Role](roles/bootstrap/README.md)
Performs initial server setup and hardening:
- Creates user accounts with SSH key authentication
- Configures passwordless sudo and security hardening
- Installs essential packages and configures UFW firewall
- Sets up Tailscale VPN for secure network access
### [Common Role](roles/common/README.md)
Provides shared configuration for all servers:
- Installs common packages (aptitude)
- Enables UFW firewall with default deny policy
- Ensures consistent base configuration across infrastructure
### [Cron Role](roles/cron/README.md)
Manages scheduled tasks and automation:
- **Warhammer RSS Feed Updater**: Daily job that generates and updates RSS feeds
- Integrates with Docker services for content generation
- Supports easy addition of new scheduled tasks
### [Docker Role](roles/docker/README.md)
The most comprehensive role, deploying 21 containerized services organized into logical categories:
- **Infrastructure**: Caddy reverse proxy, Pocket ID SSO, Dockge management
- **Development**: Gitea, ByteStash
- **Media**: Audiobookshelf, Kavita, Karakeep, Manyfold
- **Productivity**: Paperless-NGX, Baikal, Syncthing, HeyForm, Dawarich, Obsidian LiveSync
- **Communication**: GoToSocial
- **Monitoring**: Changedetection, AppriseAPI, Gotify, Cronmaster
- **Template-Driven**: All services use Jinja2 templates for consistent configuration
- **Category-Based Deployment**: Deploy services by category using Ansible tags
Role details: [bootstrap](roles/bootstrap/README.md),
[common](roles/common/README.md), [docker](roles/docker/README.md),
[cron](roles/cron/README.md).
## Usage
### Prerequisites
### Setup
1. Install Ansible and required collections:
```bash
ansible-galaxy install -r requirements.yml
```
2. Configure your inventory in `hosts.yml` with your server details
### Bootstrap a New Server
The `bin/` helpers use the virtualenv at `.venv/` in the main checkout. The
`ansible` package brings the collections used here (`community.docker`,
`community.general`, `amazon.aws`, `ansible.posix`); `boto3` is needed for
Route53.
```bash
ansible-playbook bootstrap.yml -i hosts.yml
python -m venv .venv && .venv/bin/pip install ansible boto3
.venv/bin/ansible-galaxy install -r requirements.yml
```
This will:
- Create a user account
- Install and configure Tailscale VPN
- Set up basic security
`vault_pass` and `group_vars/all/vault.yml` are gitignored and live only in
the main checkout. `bin/worktree` links them into each worktree.
### Deploy Docker Services
### Making a change
Several agents work on this repo at once, so every change follows the same
steps (see [CLAUDE.md](CLAUDE.md) for why):
```bash
ansible-playbook site.yml -i hosts.yml
bin/worktree my-change # new worktree off origin/main
# ...edit, commit...
git fetch && git rebase origin/main
git push origin HEAD:main
bin/deploy --tags <service> # deploy that exact commit
```
Deploy specific services using tags:
```bash
# Deploy by service category
ansible-playbook site.yml -i hosts.yml --tags infrastructure
ansible-playbook site.yml -i hosts.yml --tags media,productivity
# Deploy all services (skips system setup)
ansible-playbook site.yml -i hosts.yml --tags services
# Deploy individual services
ansible-playbook site.yml -i hosts.yml --tags caddy
ansible-playbook site.yml -i hosts.yml --tags pocketid
```
### Manage DNS Records
### Deploying
```bash
ansible-playbook dns.yml -i hosts.yml
bin/deploy --tags kavita # one service
bin/deploy --tags media,productivity # whole categories
bin/deploy --tags services # every service, no system setup
bin/deploy # everything in site.yml
bin/deploy --check --diff --tags caddy # dry run (allowed from any checkout)
PLAYBOOK=dns.yml bin/deploy # DNS records
```
Updates AWS Route53 DNS records for configured domains (`thesatelliteoflove.com` and `nerder.land`).
Category tags: `infrastructure`, `development`, `media`, `productivity`,
`communication`, `monitoring`. Role tags: `common`, `docker`, `cron`. Each
service has its own tag, named after its task file (e.g. `paperlessngx`,
`obsidian-livesync`); Karakeep answers to both `karakeep` and `hoarder`.
## Configuration
### Bootstrapping a new server
- Service configurations are templated in `roles/docker/templates/`
- Environment variables and secrets should be managed through Ansible Vault
- Docker Compose files are generated from Jinja2 templates
```bash
.venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass
```
## Security Notes
### Secrets
- Uses Tailscale for secure network access
- Caddy provides automatic HTTPS with Let's Encrypt
- Services are containerized for isolation
- UFW firewall rules are managed via Docker integration
All secrets live in `group_vars/all/vault.yml` under `vault_<service>` keys.
Edit them only with `bin/vault-run <command>`, never `ansible-vault edit`.
## Security notes
- UFW allows SSH from anywhere. `ufw-docker` blocks container ports from the
internet except Caddy (80/443) and Gitea SSH (222).
- Admin tools (Syncthing, Cronmaster, Apprise, CouchDB) bind to the Tailscale
IP only. Dockge has no public route.
- Pocket ID is the single sign-on for most apps. Authentik was retired on
2026-10-31.
- Paperless-ngx and GoToSocial back up their volumes daily to S3 (7-day
retention). Other services have no automated backups yet (see
[todo.md](todo.md)).
roles/bootstrap/README.md
+19 -31
View File
@@ -1,41 +1,29 @@
# Bootstrap Role
## Purpose
Performs initial server setup and hardening for new Ubuntu/Debian servers.
First-time setup for a fresh Debian/Ubuntu server. Run once, against the
server's public IP (inventory group `bootstrap`, host `netcup`), before
`site.yml` can reach it over Tailscale.
## What It Does
## What it does
- Installs `aptitude`, `curl`, `vim`, `git` and `ufw`.
- Gives the `sudo` group passwordless sudo.
- Creates the user `created_username` (set to `phil` in `bootstrap.yml`) in
the `sudo` group, with your local `~/.ssh/id_ed25519.pub` as its SSH key.
- Sets `PermitRootLogin prohibit-password` (root can still log in with a key).
- UFW: allows OpenSSH, enables the firewall with default deny.
### User Management
- Creates a new user account with sudo privileges (specified by `created_username` variable)
- Configures passwordless sudo for the sudo group
- Sets up SSH key authentication using your local `~/.ssh/id_ed25519.pub` key
- Disables root password authentication
`bootstrap.yml` then runs the `artis3n.tailscale` role to join the tailnet,
using `vault_infrastructure.tailscale_key` from the vault.
### System Packages
- Installs `aptitude` for better package management
- Installs essential packages:
- `curl` - HTTP client
- `vim` - Text editor
- `git` - Version control
- `ufw` - Uncomplicated Firewall
### Security Configuration
- Configures UFW firewall to:
- Allow SSH connections
- Enable firewall with default deny policy
- Hardens SSH configuration
## Variables Required
- `created_username`: The username to create (typically set in bootstrap.yml)
- `tailscale_key`: Tailscale authentication key (prompted during playbook run)
## Dependencies
- Requires the `artis3n.tailscale` role for VPN setup
- Requires your SSH public key at `~/.ssh/id_ed25519.pub`
## Requirements
- `ansible-galaxy install -r requirements.yml` (for `artis3n.tailscale`).
- `~/.ssh/id_ed25519.pub` on the machine running Ansible.
- Root (or another sudo user) SSH access to the new server.
## Usage
```bash
ansible-playbook bootstrap.yml -i hosts.yml
.venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass
```
This role is designed to be run once on a fresh server before deploying other services.
Once the server is on Tailscale, put its Tailscale IP in `hosts.yml` under
`docker` and use `bin/deploy` from then on.
roles/common/README.md
+8 -18
View File
@@ -1,23 +1,13 @@
# Common Role
## Purpose
Provides shared configuration and security setup that applies to all servers in the infrastructure.
Base setup applied first by `site.yml` (tag `common`).
## What It Does
### System Packages
- Installs `aptitude` for better package management and dependency resolution
- Updates package cache to ensure latest package information
### Security Configuration
- Enables UFW (Uncomplicated Firewall) with default deny policy
- Provides baseline firewall protection for all managed servers
## What it does
- Installs `aptitude` and `jq`.
- Makes sure UFW is enabled with default deny. The SSH rule comes from the
bootstrap role, and container rules from `ufw-docker` in the docker role.
## Usage
This role is automatically applied to all servers in the infrastructure as part of the main site.yml playbook. It ensures consistent base configuration across all managed systems.
## Dependencies
None - this is a foundational role that other roles can depend on.
## Notes
This role is designed to be lightweight and provide only the most essential common functionality. Server-specific configurations should be handled by dedicated roles like `docker` or `bootstrap`.
```bash
bin/deploy --tags common
```
roles/cron/README.md
+27 -26
View File
@@ -1,35 +1,36 @@
# Cron Role
## Purpose
Manages scheduled tasks and automated maintenance jobs for the infrastructure.
Cron jobs on the server (tag `cron`). "Cron jobs" in this repo always means
jobs on the server, never on the machine running Ansible.
## What It Does
## Jobs
### Warhammer RSS Feed Updater
- Copies `update_warhammer_feed.sh` script to `/usr/local/bin/` with executable permissions
- Creates a daily cron job that runs at 09:10 AM
- The script performs these actions:
1. Creates a temporary directory `/tmp/warhammer_feed`
2. Runs a custom Docker container (`git.thesatelliteoflove.com/phil/rss-warhammer`) to generate RSS feed
3. Copies the generated `warhammer_rss_feed.xml` to `/opt/stacks/caddy/site/tsol/feeds/`
| Time | User | Job |
|---|---|---|
| 08:00 daily | phil | `~/.local/bin/run_dockcheck.sh`: container updates |
| 09:10 daily | root | `/usr/local/bin/update_warhammer_feed.sh`: Warhammer RSS feed |
## Files Managed
- `/usr/local/bin/update_warhammer_feed.sh` - RSS feed update script
- Cron job: "Update Warhammer RSS Feed" (daily at 09:10)
## Dependencies
- Requires Docker to be installed and running
- Depends on the following Docker stacks being deployed:
- Custom RSS generator container at `git.thesatelliteoflove.com/phil/rss-warhammer`
- Caddy web server stack at `/opt/stacks/caddy/`
## Usage
This role is automatically applied as part of the main site.yml playbook with the `cron` tag.
### Dockcheck (container updates)
- Downloads [dockcheck](https://github.com/mag37/dockcheck) and its Gotify
notify templates into `/home/phil/.local/bin/`.
- Templates `/home/phil/.config/dockcheck.config` from `dockcheck.config.j2`
(Gotify token from `vault_dockcheck`).
- Settings: `OnlyLabel=true` and `AutoMode=true`, so containers labelled
`mag37.dockcheck.update=true` are pulled and recreated automatically; any
other container with an update gets a Gotify notice. `dawarich_db` and
`dawarich_redis` are skipped.
- `run_dockcheck.sh` logs to `/var/log/dockcheck/dockcheck.log`.
To see what needs updating by hand, run dockcheck on the server:
```bash
# Deploy only cron jobs
ansible-playbook site.yml -i hosts.yml --tags cron
ssh phil@100.70.169.99 '~/.local/bin/dockcheck.sh -n'
```
## Customization
To add additional cron jobs, create new tasks in the main.yml file following the same pattern as the Warhammer feed updater.
### Warhammer RSS feed
`update_warhammer_feed.sh` runs the `git.thesatelliteoflove.com/phil/rss-warhammer`
container to build `warhammer_rss_feed.xml` and copies it to
`/opt/stacks/caddy/site/tsol/feeds/`, where Caddy serves it on the main site.
## Adding a job
Add a `cron` task to `tasks/main.yml` (copy the script with `copy` first if it
needs one), then `bin/deploy --tags cron`.
roles/docker/README.md
+120 -197
View File
@@ -1,221 +1,144 @@
# Docker Role
## Purpose
Deploys and manages a comprehensive self-hosted infrastructure with 24 containerized services organized into logical categories, transforming a server into a personal cloud platform with authentication, media management, productivity tools, and development services.
Installs Docker and deploys all 24 services. Each service is a Docker Compose
stack in `/opt/stacks/<service>/` (Dockge lives in `/opt/dockge/`), built from
Jinja2 templates in this role.
## Architecture Overview
## What runs before the services
### Network Configuration
- **External Network**: All services connect to shared Docker network (configurable)
- **Reverse Proxy**: Caddy handles all ingress traffic with automatic HTTPS
- **Service Discovery**: Container-to-container communication using service names
- **Firewall Integration**: UFW-Docker script properly configures firewall rules
`tasks/main.yml` does the system setup, then imports each category:
### Security Features
- **Centralized SSO**: Pocket ID provides passkey OIDC authentication for most services
- **Network Isolation**: Services restricted to appropriate network segments
- **Container Hardening**: Non-root users, capability dropping, security options
- **Secret Management**: Ansible vault for sensitive configuration
- **Variable Management**: Centralized variable hierarchy using group_vars structure
1. Base packages (`python3-pip`, `virtualenv`, `duf`, …).
2. Shell setup (`shell.yml`): zsh as the login shell, Atuin (installed and
logged in once, dotfile sync on), and `bat`.
3. Docker CE from Docker's apt repository.
4. `ufw-docker.sh` copied to `/usr/local/bin/` and its rules installed (UFW is
restarted once after install).
5. The shared `lava` network (see below).
6. `/opt/stacks` and `/opt/dockge`.
## Services Deployed (Organized by Category)
## Services by category
### Infrastructure (`infrastructure/`)
- **Caddy** - Reverse proxy with automatic HTTPS (static IP: 172.20.0.5)
- **Pocket ID** - Passkey OIDC provider
- **Dockge** - Docker compose stack management UI
Tags: each category file has a category tag, each service its own tag.
### Development (`development/`)
- **Gitea** - Self-hosted Git with CI/CD runners
- **Code Server** - VS Code in the browser
- **ByteStash** - Code snippet management and organization
### Infrastructure (`infrastructure`)
| Tag | Service | Notes |
|---|---|---|
| `caddy` | Caddy | Fixed IP 172.20.0.5. Ports 80/443 (+443/udp). `Caddyfile.j2` changes restart it via a handler; the stack itself is deployed with `recreate: never`. Static sites are served from `/opt/stacks/caddy/site/`. |
| `pocketid` | Pocket ID v2 | Passkey OIDC provider at `id.`. OIDC clients are created in its web UI; their IDs and secrets go in the vault. |
| `gondulf` | Gondulf | IndieAuth server for the main site (own image from Gitea). Sends login codes by email through Resend. |
| `dockge` | Dockge | Compose UI on port 5001, managing `/opt/stacks`. |
### Media (`media/`)
- **Audiobookshelf** - Audiobook server (Pocket ID login)
- **Kavita** - E-book server (Pocket ID login)
- **Ghost** - Modern blogging platform
- **Pinchflat** - YouTube video archiving
- **Pinry** - Pinterest-like image board
- **Karakeep** - Bookmark management with AI tagging
- **Manyfold** - 3D model file organization
### Development (`development`)
| Tag | Service | Notes |
|---|---|---|
| `gitea` | Gitea + act_runner | Fixed IP 172.20.0.6 so the ufw-docker rule for SSH (host port 222) stays valid. Pocket ID login is set up in Gitea's admin UI; OpenID 2.0 login is off. |
| `bytestash` | ByteStash | Pocket ID through env vars. |
### Productivity (`productivity/`)
- **Paperless-ngx** - Document management with OCR
- **MMDL** - Task and calendar management with CalDAV integration
- **Baikal** - CalDAV/CardDAV server
- **Syncthing** - Decentralized file sync
- **Heyform** - Form builder and surveys
- **Dawarich** - Location tracking
- **Palmr** - File sharing service
- **Obsidian LiveSync** - CouchDB backend for note synchronization
### Media (`media`)
| Tag | Service | Notes |
|---|---|---|
| `audiobookshelf` | Audiobookshelf | Pocket ID set up in the app's settings. Podcasts are not used. |
| `kavita` | Kavita | Ebooks at `books.`. Also handles OPDS and KOReader sync (replaced Calibre-Web, dir2opds and kosync). Pocket ID set up in the app's settings. |
| `karakeep`, `hoarder` | Karakeep | Files are still named `hoarder*`. Web, headless Chrome and Meilisearch. Pocket ID through `hoarder-env.j2`. |
| `manyfold` | Manyfold | Pocket ID through env vars. |
### Communication (`communication/`)
- **GoToSocial** - Lightweight ActivityPub server
- **Postiz** - Social media management
### Productivity (`productivity`)
| Tag | Service | Notes |
|---|---|---|
| `paperlessngx` | Paperless-ngx | Redis, Gotenberg, Tika. Pocket ID through django-allauth (`paperlessngx.env.j2`). Daily S3 backup of the data volume. |
| `baikal` | Baikal | CalDAV/CardDAV. Caddy redirects the `.well-known` paths. |
| `syncthing` | Syncthing | Ports on the Tailscale IP only. |
| `heyform` | HeyForm | MongoDB (Percona) and KeyDB. On `forms.nerder.land`. |
| `dawarich` | Dawarich | App, Sidekiq, PostGIS, Redis. Based on the upstream production compose file. |
| `obsidian-livesync` | CouchDB for Obsidian LiveSync | Port 5984 on the Tailscale IP only. Has a restart handler. |
| `gokapi` | Gokapi | Pocket ID login. `config.json` is only written on a fresh install, because Gokapi rewrites it on upgrade. To change it, delete `/opt/stacks/gokapi/config/config.json` and redeploy. |
### Monitoring (`monitoring/`)
- **Change Detection** - Website monitoring
- **Apprise API** - Unified notifications
- **Gotify** - Self-hosted push notification service
### Communication (`communication`)
| Tag | Service | Notes |
|---|---|---|
| `gotosocial` | GoToSocial | Pocket ID through env vars. Account domain is the apex; Caddy serves WebFinger there. Daily S3 backup of its volume. |
| `listmonk` | Listmonk | Postgres 17. On `listmonk.nerder.land`. |
| `starpunk` | StarPunk | Micropub server for the main site (own image from Gitea). Logs in through Gondulf. |
## Deployment Patterns
### Monitoring (`monitoring`)
| Tag | Service | Notes |
|---|---|---|
| `changedetection` | Changedetection | With a sockpuppetbrowser for JavaScript pages. |
| `appriseapi` | Apprise API | Port 8000 on the Tailscale IP only. Other containers reach it at `http://apprise:8000`. |
| `gotify` | Gotify + iGotify assistant | Push notifications. Dockcheck sends its update notices here. |
| `cronmaster` | Cronmaster | Cron UI, port 40123 on the Tailscale IP only. |
### Standardized Service Deployment
Each service follows a consistent pattern:
1. Creates `/opt/stacks/[service-name]` directory structure
2. Generates Docker Compose file from Jinja2 template
3. Deploys using `community.docker.docker_compose_v2`
4. Configures environment variables from vault secrets
## Networking
### Template System
- **Compose Templates**: `.j2` files in `templates/` for dynamic configuration
- **Environment Templates**: Separate `.env.j2` files for services requiring environment variables
- **Variable Substitution**: Uses centralized variable hierarchy from group_vars structure
- **Domain Management**: Centralized domain and subdomain configuration
- **Network Configuration**: Standardized Docker network and IP address management
### Shared network (`lava`)
Every stack joins the external network `lava` (`172.20.0.0/16`). Docker only
hands out dynamic addresses from `172.20.128.0/17`, so `172.20.0.x` is free
for fixed IPs: Caddy (.5) and Gitea (.6). Both need fixed IPs because the
ufw-docker rules point at container IPs. The role creates the network if it
is missing and warns if it lacks the reserved range; `migrate-lava-network.yml`
fixes an old network.
## Shell Environment Setup
The role also configures the shell environment:
- **Zsh Installation**: Sets zsh as default shell
- **Atuin**: Command history sync and search
- **Bat**: Enhanced `cat` command with syntax highlighting
### Hairpinning
Containers can't reach the server's own public IP, so any app that calls
another of our domains (usually Pocket ID for OIDC) maps it to Caddy:
```yaml
extra_hosts:
- "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}"
```
### Firewall
`ufw-docker.sh` makes UFW apply to published container ports. Only Caddy
(80/tcp, 443/tcp, 443/udp) and Gitea (22/tcp, published as 222) are allowed
in from the internet; the Caddy and Gitea task files add those rules.
## Updates
Containers labelled `mag37.dockcheck.update: true` are updated automatically
each morning by dockcheck (set up in the cron role). Containers without the
label (Gondulf, StarPunk, Listmonk, HeyForm, Dockge and most databases) only
get a Gotify notice. Dawarich's database and Redis are skipped entirely.
Gondulf and StarPunk use pinned version tags.
## Adding a service
1. `templates/<service>-compose.yml.j2` (plus `-env.j2` if needed). Join the
external `{{ docker.network_name }}` network and add `extra_hosts` for any
of our domains the app calls.
2. `tasks/<category>/<service>.yml`: make `/opt/stacks/<service>`, template
the files, deploy with `community.docker.docker_compose_v2`. Use
`diff: false` or `no_log: true` on tasks that write secrets.
3. Import it in `tasks/<category>/main.yml` with a tag named after the service.
4. Subdomain in `group_vars/all/domains.yml`, route in `templates/Caddyfile.j2`,
A record in `dns.yml`.
5. Secrets under `vault_<service>` with `bin/vault-run`.
## Files
## File Organization
```
roles/docker/
├── tasks/
│ ├── main.yml # Orchestrates all deployments
│ ├── shell.yml # Shell environment setup
│ ├── infrastructure/
│ │ ├── main.yml # Infrastructure category orchestrator
│ │ ├── caddy.yml # Reverse proxy
│ │ └── pocketid.yml # Authentication
│ ├── development/
│ │ ├── main.yml # Development category orchestrator
│ │ ├── gitea.yml # Git hosting
│ │ └── bytestash.yml # Code snippets
│ ├── media/ # Media services (7 services)
│ ├── productivity/ # Productivity services (7 services)
│ ├── communication/ # Communication services (2 services)
│ └── monitoring/ # Monitoring services (3 services)
├── templates/
│ ├── [service]-compose.yml.j2 # Docker Compose templates (all templated)
│ ├── [service]-env.j2 # Environment variable templates
│ └── [service]-*.j2 # Service-specific templates
├── files/
│ ├── Caddyfile # Caddy configuration
│ ├── ufw-docker.sh # Firewall integration script
│ ├── client # Matrix well-known client file
│ └── server # Matrix well-known server file
└── handlers/
└── main.yml # Service restart handlers
│ ├── main.yml # System setup, network, category imports
│ ├── shell.yml # zsh, Atuin, bat
│ ├── infrastructure/ # caddy, pocketid, gondulf (+ dockge in main.yml)
│ ├── development/ # gitea, bytestash
│ ├── media/ # audiobookshelf, kavita, hoarder (Karakeep), manyfold
│ ├── productivity/ # paperlessngx, baikal, syncthing, heyform, dawarich, obsidian-livesync, gokapi
│ ├── communication/ # gotosocial, listmonk, starpunk
│ └── monitoring/ # changedetection, appriseapi, gotify, cronmaster
├── templates/ # <service>-compose.yml.j2, env and config templates, Caddyfile.j2
├── files/ufw-docker.sh # UFW integration for Docker
└── handlers/main.yml # restart caddy, restart obsidian-livesync
```
## Usage
### Deploy All Services
```bash
ansible-playbook site.yml -i hosts.yml --tags docker
```
### Deploy by Service Category
```bash
# Deploy entire service categories
ansible-playbook site.yml -i hosts.yml --tags infrastructure
ansible-playbook site.yml -i hosts.yml --tags development
ansible-playbook site.yml -i hosts.yml --tags media
ansible-playbook site.yml -i hosts.yml --tags productivity
ansible-playbook site.yml -i hosts.yml --tags communication
ansible-playbook site.yml -i hosts.yml --tags monitoring
# Deploy multiple categories
ansible-playbook site.yml -i hosts.yml --tags infrastructure,monitoring
```
### Deploy Individual Services
```bash
# Deploy specific services
ansible-playbook site.yml -i hosts.yml --tags pocketid
ansible-playbook site.yml -i hosts.yml --tags gitea,bytestash
ansible-playbook site.yml -i hosts.yml --tags paperlessngx
```
## Service-Specific Notes
### MMDL (Task Management)
- **URL**: https://tasks.thesatelliteoflove.com
- **Initial Setup**: Visit `/install` endpoint first to run database migrations
- **Authentication**: Integrates with the Pocket ID OIDC provider
- **Database**: Uses MySQL 8.0 with automatic schema migration
- **Features**: CalDAV integration, multiple account support, task management
## Dependencies
### System Requirements
- Docker CE installed and running
- UFW firewall configured
- DNS records pointing to the server
- Valid SSL certificates (handled automatically by Caddy)
### External Services
- **DNS**: Requires subdomains configured for each service
- **Email**: Gitea uses Resend for notifications
- **Storage**: All services persist data to `/opt/stacks/[service]/`
## Configuration
### Variable Structure
The role uses a centralized variable hierarchy in `group_vars/all/`:
- **domains.yml**: Domain and subdomain mappings for all services
- **infrastructure.yml**: Network configuration, Docker settings, and system parameters
- **vault.yml**: Encrypted secrets including API keys, passwords, and OAuth credentials
- **services.yml**: Service-specific configuration and feature flags
### Required Variables (in vault.yml)
- Authentication credentials for various services (vault_*)
- API keys for external integrations
- OAuth client secrets for SSO integration
- Database passwords and connection strings
- SMTP credentials for notifications
### Network Configuration
Services expect to be accessible via subdomains of configured domains:
- `id.thesatelliteoflove.com` - Pocket ID
- `git.thesatelliteoflove.com` - Gitea
- `books.thesatelliteoflove.com` - Kavita
- `tasks.thesatelliteoflove.com` - MMDL
- (and many more...)
## Monitoring & Management
### Operational Features
- Automatic container restart policies
- Health checks for database services
- Centralized logging and monitoring
- Backup-ready data structure in `/opt/stacks/`
## Security Considerations
### Network Security
- UFW-Docker integration for proper firewall rules
- Services isolated to appropriate network segments
- Restricted access for sensitive tools (Stirling PDF)
### Authentication
- Centralized SSO through Pocket ID for most services
- OAuth integration where supported
- Secure secret management through Ansible vault
### Container Security
- Non-root container execution (UID/GID 1000:1000)
- Security options: `no-new-privileges: true`
- Capability dropping and minimal permissions
## Troubleshooting
### Common Issues
- **Database Connection**: Ensure MySQL containers use proper authentication plugins
- **OAuth Discovery**: Check issuer URLs don't have trailing slashes
- **Migration Failures**: Visit service `/install` endpoints for database setup
- **Network Issues**: Verify containers are on the same Docker network
- **OIDC login fails with a connection error:** the container is missing the
`extra_hosts` entry for `id.thesatelliteoflove.com`.
- **OIDC discovery fails:** check the issuer URL has no trailing slash.
- **Caddy fails with "Address already in use":** another container took
172.20.0.5; the network lacks the reserved range. Run
`migrate-lava-network.yml`.
- **Gitea SSH stops working after a recreate:** Gitea lost its fixed IP;
redeploy with `--tags gitea` to restore it and the firewall rule.
todo.md
+45 -128
View File
@@ -1,141 +1,58 @@
# Infrastructure Improvements TODO
## High Priority (Quick Wins)
Current state (2026-10-31): 24 services in 6 categories, all compose files
templated, Pocket ID as SSO (Authentik retired), multi-agent guard rails
(`bin/worktree`, `bin/deploy`, `bin/vault-run`, `deploy-guard.yml`).
### 1. Split the massive docker role ✅ COMPLETED
- **Current Issue**: `roles/docker/tasks/main.yml` has 20+ services in one file (176 lines)
- **Solution**: Break into logical service groups:
```
roles/docker/tasks/
├── main.yml (orchestrator)
├── infrastructure/ (caddy, pocketid, dockge)
├── development/ (gitea, codeserver, bytestash)
├── media/ (audiobookshelf, kavita, ghost, pinchflat, pinry, hoarder, manyfold)
├── productivity/ (paperless, baikal, syncthing, mmdl, heyform, dawarich, palmr, obsidian-livesync)
├── communication/ (gotosocial, postiz)
└── monitoring/ (changedetection, appriseapi, gotify)
```
- **COMPLETED**: All services organized into logical categories with category-level tags
## Done
- Split the docker role into categories with category and service tags.
- Central variables in `group_vars/all/` (`domains.yml`, `infrastructure.yml`,
`vault.yml`).
- Converted every static compose file to a template.
- Removed unused services, most recently Authentik, Calibre-Web, dir2opds,
kosync, Sneaky Klaus and Audiobookshelf podcasts.
- Reserved address range on the `lava` network so fixed IPs can't be taken.
- Automatic container updates with dockcheck for labelled containers.
### 2. Standardize variable management ✅ COMPLETED
- **Current Issue**: Secrets in single encrypted file, no clear variable hierarchy
- **Solution**: Create proper variable structure:
```
group_vars/
├── all/
│ ├── domains.yml (domain and subdomain mappings)
│ ├── infrastructure.yml (network config, Docker settings)
│ ├── vault.yml (encrypted secrets with vault_ prefix)
│ └── services.yml (service configuration flags)
```
- **COMPLETED**: Implemented comprehensive variable hierarchy, updated all templates to use centralized variables, fixed service tag isolation
## Open
### 3. Template consolidation ✅ PARTIALLY COMPLETED
- **Current Issue**: Many compose templates repeat patterns, some services used static files
- **Solution**: Create reusable template includes with standard service template structure
- **COMPLETED**: Converted all static compose files (caddy, dockge, hoarder) to Jinja2 templates
- **REMAINING**: Create reusable template patterns for common configurations
### Reusable template patterns
Many compose templates repeat the same network, restart and label blocks.
Pull the shared parts into includes or macros.
## Security & Reliability
### Health checks
Only Pocket ID, Gondulf, Dawarich, Listmonk and Syncthing define health checks
in their templates. Add them to the rest where the image supports it.
### 4. Add health checks
- **Issue**: Most services lack proper healthcheck configurations in compose templates
- **Solution**: Implement comprehensive health monitoring with standardized healthcheck patterns
### Backups
Only Paperless-ngx and GoToSocial back up (daily to S3 with
`offen/docker-volume-backup`, 7-day retention). Still to do:
- Database dumps for the Postgres/MongoDB/CouchDB services.
- Volume backups for the file-based services (Kavita, Audiobookshelf, Gitea,
Pocket ID, …).
- A tested restore.
### 5. Implement backup strategy
- **Issue**: No automated backups for 25 services and their data
- **Solution**: Add backup role with:
- Database dumps for PostgreSQL services
- Volume backups for file-based services
- Rotation policies
- Restoration testing
### Network segmentation
Every service shares the `lava` network. Consider separate frontend, backend
and database networks.
### 6. Network segmentation
- **Issue**: All services share one Docker network
- **Solution**: Separate into:
- `frontend` (Public-facing services)
- `backend` (Internal services only)
- `database` (Database access only)
### Security hardening
- Add `no-new-privileges` and dropped capabilities where images allow it
(only Manyfold has them now). Cronmaster runs as root.
- fail2ban (or Caddy rate limits) in front of login pages.
### 7. Security hardening
- Remove unnecessary `user: root` from services
- Add security contexts to all containers
- Implement least-privilege access patterns
- Add fail2ban for authentication services
### CI with Gitea Actions
Syntax check and `--check` runs on push, using the existing Gitea runner.
## Automation Opportunities
### Configuration drift detection
Catch manual changes on the server, e.g. a scheduled `bin/deploy --check`.
### 8. CI/CD with Gitea Actions
- Leverage self-hosted Gitea for:
- Ansible syntax validation
- Service configuration testing
- Automated deployment triggers
- Rollback capabilities
### Service start order
Apps that use Pocket ID don't wait for it to be up.
### 9. Configuration drift detection
- Add validation tasks to catch manual changes
- Implement configuration validation with proper assertions
### 10. Service dependency management
- **Issue**: Some services depend on Pocket ID SSO but no startup ordering
- **Solution**: Implement dependency checking and startup ordering
### 11. Ansible best practices
- Replace deprecated `apt_key` with proper patterns
- Use `ansible.builtin` FQCN consistently
- Add `check_mode` support
- Implement proper idempotency checks
### 12. Documentation automation
- Auto-generate service inventory
- Create service documentation templates
- Implement automated documentation updates
## Implementation Roadmap
### Week 1: Foundation
- [x] Document improvements in todo.md
- [x] Reorganize docker role structure
- [x] Convert static compose files to templates
- [x] Remove unused services (beaver, grist, stirlingpdf, tasksmd, redlib)
- [x] Clean up templates and files directories
- [x] Implement variable hierarchy
- [ ] Create reusable template patterns
### Week 2: Security & Monitoring
- [ ] Add health checks
- [ ] Implement backup strategy
- [ ] Security hardening
### Week 3: Automation
- [ ] CI/CD pipeline setup
- [ ] Configuration validation
- [ ] Documentation automation
### Week 4: Advanced Features
- [ ] Network segmentation
- [ ] Dependency management
- [ ] Monitoring dashboard
## Completed Work Summary
### ✅ Major Accomplishments
- **Docker Role Reorganization**: Split monolithic 176-line main.yml into 6 logical service categories
- **Template Standardization**: Converted all static compose files to Jinja2 templates
- **Service Cleanup**: Removed 5 unused/broken services (beaver, grist, stirlingpdf, tasksmd, redlib)
- **Category-Based Deployment**: Can now deploy services by category using tags (infrastructure, media, etc.)
- **Variable Management**: Implemented comprehensive centralized variable hierarchy with proper secret organization
- **Service Tag Isolation**: Fixed service tags to deploy individual services only (not entire categories)
- **Documentation Updates**: Updated all README files and CLAUDE.md to reflect new architecture
### 📊 Current Stats
- **25 active services** organized into 6 categories
- **100% templated** compose files (no static files)
- **6 service directories** for logical organization
- **Clean file structure** with only essential static files
## Notes
- Current architecture is solid and much better organized for long-term maintainability
- Focus on high-impact, low-effort improvements first
- Leverage existing infrastructure (Gitea, Pocket ID) for automation
- Template-driven approach enables future dynamic configuration
### Ansible clean-up
- Replace the deprecated `apt_key` task for Docker's repo (it also fetches the
Ubuntu key for a Debian repo).
- Use `ansible.builtin.` names everywhere.
- The Dockge compose file is templated twice (`tasks/main.yml` and
`tasks/infrastructure/main.yml`).