phil/ansible
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bring the docs up to date with the running stack

Browse Source 
The READMEs, CLAUDE.md and todo.md still described Authentik, services
removed long ago (Ghost, MMDL, Postiz, Pinchflat...) and two servers. Now
they list the 24 services actually deployed (checked against docker ps),
explain that netcup and docker-01 are one machine, document Gondulf,
StarPunk, Listmonk, Gokapi and dockcheck, and use bin/deploy throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Phil
2026-10-31 11:30:50 -06:00
co-authored by Claude Opus 5.5
parent 0282debf07
commit 0c14ee5b1d
8 changed files with 426 additions and 639 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.local.md
+1 -1
View File
@@ -4,4 +4,4 @@
- Whenever I talk about cron jobs I am referring to cron jobs on the remote servers managed by ansible, never the local machine - Whenever I talk about cron jobs I am referring to cron jobs on the remote servers managed by ansible, never the local machine
- never use secrets.enc - never use secrets.enc
- all secrets go in vault.yml, never secrets.enc, never some random file you want to create, only ever vault.yml. you decrypt vault.yml with vault_pass - all secrets go in vault.yml, never secrets.enc, never some random file you want to create, only ever vault.yml. you decrypt vault.yml with vault_pass
- Never use ansible-vault edit. always decrypt, make the changes, then encrypt - Never use ansible-vault edit. Edit vault.yml only with bin/vault-run <command> (it decrypts, runs your edit, checks and re-encrypts)
CLAUDE.md
+89 -118
View File
@@ -4,9 +4,13 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
## Overview ## Overview
This is a personal infrastructure Ansible playbook that automates deployment and management of 21 self-hosted Docker services across two domains (`thesatelliteoflove.com` and `nerder.land`). The setup uses Tailscale VPN for secure networking and Caddy for reverse proxy with automated HTTPS. Personal infrastructure as Ansible: 24 self-hosted Docker services on one
server, across `thesatelliteoflove.com` and `nerder.land`. Caddy is the
reverse proxy (automatic HTTPS), Pocket ID is the passkey SSO, and Tailscale
is how Ansible reaches the server. See `README.md` for the service list.
**Important**: Always review `DEPLOYMENT_LEARNINGS.md` when working on this repository for lessons learned and troubleshooting guidance. If a local `DEPLOYMENT_LEARNINGS.md` exists (it's gitignored), read it for
past lessons.
## Working alongside other agents (read first) ## Working alongside other agents (read first)
@@ -41,156 +45,123 @@ other. Follow these steps:
## Common Commands ## Common Commands
### Initial Setup
```bash ```bash
# Install Ansible dependencies bin/worktree <branch> # start a task (see above)
ansible-galaxy install -r requirements.yml bin/deploy --tags <service> # deploy one service
bin/deploy --tags media,productivity # deploy categories
bin/deploy --tags services # every service, no system setup
bin/deploy --tags common,cron # base setup and cron jobs only
bin/deploy --check --diff --tags <tag> # dry run, allowed from any checkout
PLAYBOOK=dns.yml bin/deploy # Route53 records
bin/vault-run <command> # edit secrets ($VAULT_PLAIN)
# Bootstrap new server (creates user, installs Tailscale, security hardening) # First-time setup of a new server (public IP, before Tailscale)
ansible-playbook bootstrap.yml -i hosts.yml .venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass
# Deploy all Docker services
ansible-playbook site.yml -i hosts.yml
# Update DNS records in AWS Route53
ansible-playbook dns.yml -i hosts.yml
``` ```
### Service Management The venv (`.venv/`), `vault_pass` and `group_vars/all/vault.yml` live in the
```bash main checkout; the `bin/` helpers find them from any worktree.
# Deploy specific services using tags (now properly isolated)
ansible-playbook site.yml -i hosts.yml --tags caddy --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags pocketid --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags gitea --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags docker --vault-password-file vault_pass # entire docker role
ansible-playbook site.yml -i hosts.yml --tags services --vault-password-file vault_pass # all services only
# Deploy services by category
ansible-playbook site.yml -i hosts.yml --tags infrastructure --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags media,productivity --vault-password-file vault_pass
ansible-playbook site.yml -i hosts.yml --tags development,monitoring --vault-password-file vault_pass
# Deploy only infrastructure components
ansible-playbook site.yml -i hosts.yml --tags common,cron --vault-password-file vault_pass
```
## Architecture ## Architecture
### Host Configuration ### Host
- **Bootstrap Host** (`netcup`): 152.53.36.98 - Initial server setup target One server, `cogitator-01` (netcup VPS), listed twice in `hosts.yml`:
- **Docker Host** (`docker-01`): 100.70.169.99 - Main service deployment via Tailscale - `netcup` (group `bootstrap`): public IP 152.53.36.98, used only by `bootstrap.yml`.
- `docker-01` (group `docker`): Tailscale IP 100.70.169.99, used by `site.yml`.
### Role Structure DNS A records point at the public IP; Caddy on the same machine routes by
- **bootstrap**: Initial server hardening, user creation, Tailscale VPN setup subdomain.
- **common**: Basic system configuration, UFW firewall management
- **docker**: Comprehensive service deployment (21 containerized applications, organized by category)
- **cron**: Scheduled task management (currently Warhammer RSS feed generation)
### Docker Role Organization (Reorganized into Logical Categories) ### Roles (`site.yml` runs common, docker, cron)
The docker role is now organized into logical service groups under `roles/docker/tasks/`: - **bootstrap**: user, SSH key, passwordless sudo, UFW, then `artis3n.tailscale`.
- **common**: `aptitude`, `jq`, UFW on with default deny.
- **docker**: Docker CE, shell setup (zsh, Atuin, bat), ufw-docker, the `lava`
network, and every service.
- **cron**: dockcheck (08:00, auto-updates labelled containers, Gotify notices)
and the Warhammer RSS feed (09:10).
- **infrastructure/**: Core platform components ### Docker role categories (`roles/docker/tasks/<category>/`)
- Caddy (reverse proxy), Pocket ID (passkey SSO), Dockge (container management) - **infrastructure/**: Caddy, Pocket ID, Gondulf (IndieAuth), Dockge
- **development/**: Development and collaboration tools - **development/**: Gitea (+ act_runner), ByteStash
- Gitea, ByteStash - **media/**: Audiobookshelf, Kavita, Karakeep (files named `hoarder`), Manyfold
- **media/**: Content creation and consumption - **productivity/**: Paperless-ngx, Baikal, Syncthing, HeyForm, Dawarich, Obsidian LiveSync, Gokapi
- Audiobookshelf, Kavita, Karakeep (formerly Hoarder), Manyfold - **communication/**: GoToSocial, Listmonk, StarPunk (Micropub)
- **productivity/**: Personal organization and document management - **monitoring/**: Changedetection, Apprise API, Gotify (+ iGotify), Cronmaster
- Paperless-NGX, Baikal (CalDAV/CardDAV), Syncthing, Heyform, Dawarich, Obsidian LiveSync
- **communication/**: Social media
- GoToSocial (Fediverse)
- **monitoring/**: System monitoring and alerts
- Changedetection, AppriseAPI, Gotify, Cronmaster
### Variable Management Each category has a tag (`infrastructure`, `media`, …) and each service its
**Critical**: This infrastructure uses a centralized variable hierarchy in `group_vars/all/`: own tag; `services` covers all of them. Karakeep has both `karakeep` and
`hoarder` tags.
- **domains.yml**: Domain and subdomain mappings (use `{{ subdomains.service }}`) ### Variables (`group_vars/all/`)
- **infrastructure.yml**: Network configuration, Docker settings (`{{ docker.network_name }}`, `{{ docker.hairpin_ip }}`) - **domains.yml**: `primary_domain`, `secondary_domain`, `subdomains.<service>`, email domains.
- **vault.yml**: Encrypted secrets with `vault_` prefix - **infrastructure.yml**: `docker.*` (network name, subnet, fixed IPs), `smtp.*` (Resend), `network.*` (public and Tailscale IPs).
- **services.yml**: Service-specific configuration and feature flags - **vault.yml**: encrypted secrets, one `vault_<service>` dict per service.
**Important**: All templates use variables instead of hardcoded values. Never hardcode domains, IPs, or secrets. Never hardcode domains, IPs or secrets in templates; use these variables.
### Data Structure ### Data
- All service data stored in `/opt/stacks/[service-name]/` on docker host - Each service lives in `/opt/stacks/<service>/` on the server (Dockge in `/opt/dockge/`).
- Docker Compose files generated from Jinja2 templates in `roles/docker/templates/` - Compose files come from `roles/docker/templates/<service>-compose.yml.j2`; env and config files from `<service>-env.j2` / `.env.j2` / `-config.json.j2`.
- Environment files templated for services requiring configuration - Retired services' final backups are in `/opt/stacks/_retired/`.
- All configurations use centralized variables for consistency
## Key Implementation Details ## Key Implementation Details
### Template-Driven Configuration ### Adding or changing a service
The docker role uses Jinja2 templates exclusively for all services. When modifying services: - Template: `roles/docker/templates/<service>-compose.yml.j2`.
- Update templates in `roles/docker/templates/[service]-compose.yml.j2` - Tasks: `roles/docker/tasks/<category>/<service>.yml`, imported with a tag in that category's `main.yml`.
- Environment files use `.env.j2` templates where needed - Public access: subdomain in `domains.yml`, route in `Caddyfile.j2`, A record in `dns.yml`.
- Task files organized by category in `roles/docker/tasks/[category]/[service].yml` - Use `diff: false` / `no_log: true` on tasks that write secrets.
- All services now use templated configurations (no static compose files) - Add the label `mag37.dockcheck.update: true` if dockcheck may update it unattended.
### DNS Management ### DNS
The `dns.yml` playbook manages AWS Route53 records for both domains. All subdomains point to the netcup server (152.53.36.98), with Caddy handling internal routing to the docker host via Tailscale. `dns.yml` manages Route53 A records for both domains with a Route53-only IAM
user (`vault_aws_dns`). It also deletes the records listed under
`removed_records`; add a retired service's subdomain there.
### Security Architecture ### Security
- Tailscale provides secure networking between management and service hosts - Caddy terminates TLS with Let's Encrypt.
- Services are network-isolated using Docker - `ufw-docker` makes UFW govern container ports. Only Caddy (80/443) and
- Caddy handles SSL termination with automatic Let's Encrypt certificates Gitea SSH (host port 222) are allowed in from the internet.
- UFW firewall managed through Docker integration script - Admin UIs (Syncthing, Cronmaster, Apprise, CouchDB) bind to the Tailscale IP only.
### Service Dependencies ### SSO (Pocket ID)
Most services log in through Pocket ID (OIDC, passkeys) at `{{ subdomains.pocketid }}`. When deploying new services, consider: Pocket ID (`id.thesatelliteoflove.com`) replaced Authentik on 2026-10-31.
- Whether SSO integration is needed Apps that log in with it: Gitea, ByteStash, Audiobookshelf, Kavita, Karakeep,
- Caddy routing configuration for subdomain access Manyfold, Paperless-ngx, Gokapi, GoToSocial. Some read the client from env
- Network connectivity requirements within Docker stack vars; Gitea, Audiobookshelf and Kavita store it in their own database (set in
- Hairpinning fixes for internal service-to-service communication their admin UI). OIDC clients are created in Pocket ID's UI and their ID and
secret go in the vault as `vault_<service>.pocketid`. StarPunk logs in with
IndieAuth through Gondulf instead.
### Shared Docker Network (`lava`) ### Shared Docker Network (`lava`)
Created by the docker role as `{{ docker.subnet }}` with `ip_range: {{ docker.ip_range }}`. Docker only hands out dynamic addresses from that range, so `172.20.0.x` is reserved for fixed IPs: Caddy (`docker.hairpin_ip`, .5) and Gitea (`docker.gitea_ip`, .6). Both need fixed IPs because ufw-docker firewall rules point at container IPs. A network without the range (the 2024 original) is migrated once with `migrate-lava-network.yml`. Created by the docker role as `{{ docker.subnet }}` with `ip_range: {{ docker.ip_range }}`. Docker only hands out dynamic addresses from that range, so `172.20.0.x` is reserved for fixed IPs: Caddy (`docker.hairpin_ip`, .5) and Gitea (`docker.gitea_ip`, .6). Both need fixed IPs because ufw-docker firewall rules point at container IPs. A network without the range (the 2024 original) is migrated once with `migrate-lava-network.yml`.
### Hairpinning Resolution ### Hairpinning
Services inside Docker containers cannot reach external domains that resolve to the same server. Fix by adding `extra_hosts` mappings: Containers can't reach domains that resolve to this server's public IP. Map
any of our domains a container calls to Caddy with `extra_hosts`:
```yaml ```yaml
extra_hosts: extra_hosts:
- "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}" - "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}"
- "{{ subdomains.cal }}:{{ docker.hairpin_ip }}"
``` ```
Common domains requiring hairpinning fixes: Most often needed for `subdomains.pocketid` (OIDC); also `subdomains.git`
- `{{ subdomains.pocketid }}` (Pocket ID SSO) (Gitea itself), `subdomains.gondulf` and `primary_domain` (StarPunk).
- `{{ subdomains.cal }}` (Baikal CalDAV)
- Any service domain the container needs to communicate with
**Note**: Use variables instead of hardcoded values for maintainability. ### Template pattern
```yaml
networks:
default:
external: true
name: "{{ docker.network_name }}"
```
### Service-Specific Reference Configurations ### Service-Specific Notes
- **Dawarich**: Based on production compose file at https://github.com/Freika/dawarich/blob/master/docker/docker-compose.production.yml - **Dawarich**: based on https://github.com/Freika/dawarich/blob/master/docker/docker-compose.production.yml
- **Gokapi**: `config.json` is only written on a fresh install (`force: false`); delete it on the server to apply template changes.
- **Caddy**: Caddyfile changes restart it via a handler; the stack uses `recreate: never`.
## Service Memories ## Service Memories
- karakeep (formerly called hoarder) is deployed with both 'hoarder' and 'karakeep' tags for backward compatibility - karakeep (formerly called hoarder) is deployed with both 'hoarder' and 'karakeep' tags for backward compatibility
- whenever i ask you what containers need updates, run dockcheck and return a list of containers needing updates - whenever i ask you what containers need updates, run dockcheck and return a list of containers needing updates
- when i ask for the status container updates i want you to run dockcheck on the docker host https://github.com/mag37/dockcheck?ref=selfh.st - when i ask for the status container updates i want you to run dockcheck on the docker host https://github.com/mag37/dockcheck?ref=selfh.st
## Variable Management Implementation Notes
**Major Infrastructure Update**: Variable management system was implemented to replace all hardcoded values with centralized variables.
### Key Changes Made:
- Created comprehensive `group_vars/all/` structure
- Updated all Docker Compose templates to use variables
- Fixed service tag isolation (individual service tags now deploy only that service)
- Standardized domain and network configuration
- Organized secrets by service with consistent `vault_` prefix
### Service Tag Fix:
**Critical**: Service tags are properly isolated. Individual service tags deploy only that service. The `services` tag runs all service tasks across all categories.
### Template Pattern:
All templates now follow this pattern:
```yaml
# Use variables, not hardcoded values
networks:
default:
external: true
name: "{{ docker.network_name }}"
extra_hosts:
- "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}"
```
README.md
+117 -120
View File
@@ -1,157 +1,154 @@
# Personal Infrastructure Ansible Playbook # Personal Infrastructure Ansible Playbook
This Ansible playbook automates the setup and management of a personal self-hosted infrastructure running Docker containers for various services. Ansible that builds and runs a personal self-hosted stack: 24 Docker services
on one server, served on `thesatelliteoflove.com` and `nerder.land`, with
Caddy for HTTPS and Pocket ID (passkeys) for single sign-on.
## Overview ## The server
The playbook manages two main environments: There is one server (hostname `cogitator-01`, a netcup VPS). The inventory
- **Bootstrap server** (`netcup`): Initial server setup with Tailscale VPN lists it twice, under two names:
- **Docker server** (`docker-01`): Main application server running containerized services
## Services Deployed | Inventory name | Address | Used by |
|---|---|---|
| `netcup` (group `bootstrap`) | 152.53.36.98 (public IP) | `bootstrap.yml`, for first-time setup before Tailscale is up |
| `docker-01` (group `docker`) | 100.70.169.99 (Tailscale IP) | `site.yml`, for everything after that |
The Docker role deploys and manages 21 self-hosted services organized into logical categories: All public DNS records point at the public IP. Caddy on the same machine
takes ports 80/443 and passes each subdomain to the right container.
### Infrastructure ## Services
- **Caddy** (Reverse proxy with automatic HTTPS)
- **Pocket ID** (passkey SSO/Identity Provider)
- **Dockge** (Container management)
### Development | Category | Service | Address | Login |
- **Gitea** (Git repository hosting) |---|---|---|---|
- **ByteStash** (Code snippet management) | Infrastructure | Caddy (reverse proxy, static sites) | — | — |
| | Pocket ID (passkey OIDC provider) | id.thesatelliteoflove.com | passkeys |
| | Gondulf (IndieAuth for the main site) | gondulf.thesatelliteoflove.com | email code |
| | Dockge (compose stack manager) | port 5001, no public route | local |
| Development | Gitea + Actions runner | git.thesatelliteoflove.com (SSH on port 222) | Pocket ID |
| | ByteStash (code snippets) | snippets.thesatelliteoflove.com | Pocket ID |
| Media | Audiobookshelf | audio.thesatelliteoflove.com | Pocket ID |
| | Kavita (ebooks, OPDS, KOReader sync) | books.thesatelliteoflove.com | Pocket ID |
| | Karakeep (bookmarks, formerly Hoarder) | bookmarks.thesatelliteoflove.com | Pocket ID |
| | Manyfold (3D models) | models.thesatelliteoflove.com | Pocket ID |
| Productivity | Paperless-ngx (documents) | paper.thesatelliteoflove.com | Pocket ID |
| | Baikal (CalDAV/CardDAV) | cal.thesatelliteoflove.com | local |
| | Syncthing | Tailscale only, port 8384 | local |
| | HeyForm (forms) | forms.nerder.land | local |
| | Dawarich (location history) | loclog.thesatelliteoflove.com | local |
| | Obsidian LiveSync (CouchDB) | Tailscale only, port 5984 | local |
| | Gokapi (file sharing) | gokapi.thesatelliteoflove.com | Pocket ID |
| Communication | GoToSocial (fediverse) | social.thesatelliteoflove.com | Pocket ID |
| | Listmonk (mailing lists) | listmonk.nerder.land | local |
| | StarPunk (Micropub for the main site) | starpunk.thesatelliteoflove.com | IndieAuth |
| Monitoring | Changedetection | watcher.thesatelliteoflove.com | local |
| | Apprise API (notifications) | Tailscale only, port 8000 | — |
| | Gotify + iGotify assistant | gotify.thesatelliteoflove.com, gotify-assistant.thesatelliteoflove.com | local |
| | Cronmaster (cron UI) | Tailscale only, port 40123 | local |
### Media "Tailscale only" means the port is bound to the Tailscale IP and has no public
- **Audiobookshelf** (Audiobook server) address.
- **Kavita** (E-book server)
- **Karakeep** (Bookmark manager, formerly Hoarder)
- **Manyfold** (3D model organizer)
### Productivity Caddy also serves static sites from `/opt/stacks/caddy/site/` (the files are
- **Paperless-NGX** (Document management) not managed by Ansible): the apex `thesatelliteoflove.com` (plus WebFinger for
- **Baikal** (CalDAV/CardDAV server) the fediverse and Tailscale login), `pdg.thesatelliteoflove.com`,
- **Syncthing** (File synchronization) `nerder.land` and `repair.nerder.land`. `repaircafe.nerder.land` redirects to
- **HeyForm** (Form builder) the Listmonk signup form.
- **Dawarich** (Location tracking)
- **Obsidian LiveSync** (Note synchronization)
### Communication ## Layout
- **GoToSocial** (Fediverse/Mastodon)
### Monitoring
- **Changedetection** (Website change monitoring)
- **AppriseAPI** (Notification service)
- **Gotify** (Push notifications)
- **Cronmaster** (Scheduled container tasks)
## Structure
``` ```
├── site.yml # Main playbook ├── site.yml # Main playbook: common, docker and cron roles
├── bootstrap.yml # Server bootstrap playbook ├── bootstrap.yml # First-time server setup (user, SSH key, UFW, Tailscale)
├── dns.yml # AWS Route53 DNS management ├── dns.yml # Route53 A records (adds current ones, removes retired ones)
├── hosts.yml # Inventory file ├── deploy-guard.yml # Stops real runs from anything but a clean origin/main
├── requirements.yml # External role dependencies ├── migrate-lava-network.yml # One-off: recreate the shared docker network
├── hosts.yml # Inventory
├── requirements.yml # Galaxy roles (artis3n.tailscale)
├── bin/
│ ├── worktree # Start a task in its own git worktree
│ ├── deploy # Run a playbook while holding the deploy lock
│ └── vault-run # Edit vault.yml safely
├── group_vars/all/
│ ├── domains.yml # Domains and subdomain map
│ ├── infrastructure.yml # Docker network, IPs, SMTP, paths
│ └── vault.yml # Encrypted secrets (gitignored)
└── roles/ └── roles/
├── bootstrap/ # Initial server setup ├── bootstrap/ # First-time setup
├── common/ # Common server configuration ├── common/ # Base packages and UFW
├── cron/ # Scheduled tasks ├── docker/ # Docker, shell setup and every service
└── docker/ # Docker services deployment └── cron/ # Host cron jobs (dockcheck, Warhammer feed)
``` ```
## Roles Documentation Role details: [bootstrap](roles/bootstrap/README.md),
[common](roles/common/README.md), [docker](roles/docker/README.md),
Each role has detailed documentation in its respective directory: [cron](roles/cron/README.md).
### [Bootstrap Role](roles/bootstrap/README.md)
Performs initial server setup and hardening:
- Creates user accounts with SSH key authentication
- Configures passwordless sudo and security hardening
- Installs essential packages and configures UFW firewall
- Sets up Tailscale VPN for secure network access
### [Common Role](roles/common/README.md)
Provides shared configuration for all servers:
- Installs common packages (aptitude)
- Enables UFW firewall with default deny policy
- Ensures consistent base configuration across infrastructure
### [Cron Role](roles/cron/README.md)
Manages scheduled tasks and automation:
- **Warhammer RSS Feed Updater**: Daily job that generates and updates RSS feeds
- Integrates with Docker services for content generation
- Supports easy addition of new scheduled tasks
### [Docker Role](roles/docker/README.md)
The most comprehensive role, deploying 21 containerized services organized into logical categories:
- **Infrastructure**: Caddy reverse proxy, Pocket ID SSO, Dockge management
- **Development**: Gitea, ByteStash
- **Media**: Audiobookshelf, Kavita, Karakeep, Manyfold
- **Productivity**: Paperless-NGX, Baikal, Syncthing, HeyForm, Dawarich, Obsidian LiveSync
- **Communication**: GoToSocial
- **Monitoring**: Changedetection, AppriseAPI, Gotify, Cronmaster
- **Template-Driven**: All services use Jinja2 templates for consistent configuration
- **Category-Based Deployment**: Deploy services by category using Ansible tags
## Usage ## Usage
### Prerequisites ### Setup
1. Install Ansible and required collections: The `bin/` helpers use the virtualenv at `.venv/` in the main checkout. The
```bash `ansible` package brings the collections used here (`community.docker`,
ansible-galaxy install -r requirements.yml `community.general`, `amazon.aws`, `ansible.posix`); `boto3` is needed for
``` Route53.
2. Configure your inventory in `hosts.yml` with your server details
### Bootstrap a New Server
```bash ```bash
ansible-playbook bootstrap.yml -i hosts.yml python -m venv .venv && .venv/bin/pip install ansible boto3
.venv/bin/ansible-galaxy install -r requirements.yml
``` ```
This will: `vault_pass` and `group_vars/all/vault.yml` are gitignored and live only in
- Create a user account the main checkout. `bin/worktree` links them into each worktree.
- Install and configure Tailscale VPN
- Set up basic security
### Deploy Docker Services ### Making a change
Several agents work on this repo at once, so every change follows the same
steps (see [CLAUDE.md](CLAUDE.md) for why):
```bash ```bash
ansible-playbook site.yml -i hosts.yml bin/worktree my-change # new worktree off origin/main
# ...edit, commit...
git fetch && git rebase origin/main
git push origin HEAD:main
bin/deploy --tags <service> # deploy that exact commit
``` ```
Deploy specific services using tags: ### Deploying
```bash
# Deploy by service category
ansible-playbook site.yml -i hosts.yml --tags infrastructure
ansible-playbook site.yml -i hosts.yml --tags media,productivity
# Deploy all services (skips system setup)
ansible-playbook site.yml -i hosts.yml --tags services
# Deploy individual services
ansible-playbook site.yml -i hosts.yml --tags caddy
ansible-playbook site.yml -i hosts.yml --tags pocketid
```
### Manage DNS Records
```bash ```bash
ansible-playbook dns.yml -i hosts.yml bin/deploy --tags kavita # one service
bin/deploy --tags media,productivity # whole categories
bin/deploy --tags services # every service, no system setup
bin/deploy # everything in site.yml
bin/deploy --check --diff --tags caddy # dry run (allowed from any checkout)
PLAYBOOK=dns.yml bin/deploy # DNS records
``` ```
Updates AWS Route53 DNS records for configured domains (`thesatelliteoflove.com` and `nerder.land`). Category tags: `infrastructure`, `development`, `media`, `productivity`,
`communication`, `monitoring`. Role tags: `common`, `docker`, `cron`. Each
service has its own tag, named after its task file (e.g. `paperlessngx`,
`obsidian-livesync`); Karakeep answers to both `karakeep` and `hoarder`.
## Configuration ### Bootstrapping a new server
- Service configurations are templated in `roles/docker/templates/` ```bash
- Environment variables and secrets should be managed through Ansible Vault .venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass
- Docker Compose files are generated from Jinja2 templates ```
## Security Notes ### Secrets
- Uses Tailscale for secure network access All secrets live in `group_vars/all/vault.yml` under `vault_<service>` keys.
- Caddy provides automatic HTTPS with Let's Encrypt Edit them only with `bin/vault-run <command>`, never `ansible-vault edit`.
- Services are containerized for isolation
- UFW firewall rules are managed via Docker integration ## Security notes
- UFW allows SSH from anywhere. `ufw-docker` blocks container ports from the
internet except Caddy (80/443) and Gitea SSH (222).
- Admin tools (Syncthing, Cronmaster, Apprise, CouchDB) bind to the Tailscale
IP only. Dockge has no public route.
- Pocket ID is the single sign-on for most apps. Authentik was retired on
2026-10-31.
- Paperless-ngx and GoToSocial back up their volumes daily to S3 (7-day
retention). Other services have no automated backups yet (see
[todo.md](todo.md)).
roles/bootstrap/README.md
+19 -31
View File
@@ -1,41 +1,29 @@
# Bootstrap Role # Bootstrap Role
## Purpose First-time setup for a fresh Debian/Ubuntu server. Run once, against the
Performs initial server setup and hardening for new Ubuntu/Debian servers. server's public IP (inventory group `bootstrap`, host `netcup`), before
`site.yml` can reach it over Tailscale.
## What It Does ## What it does
- Installs `aptitude`, `curl`, `vim`, `git` and `ufw`.
- Gives the `sudo` group passwordless sudo.
- Creates the user `created_username` (set to `phil` in `bootstrap.yml`) in
the `sudo` group, with your local `~/.ssh/id_ed25519.pub` as its SSH key.
- Sets `PermitRootLogin prohibit-password` (root can still log in with a key).
- UFW: allows OpenSSH, enables the firewall with default deny.
### User Management `bootstrap.yml` then runs the `artis3n.tailscale` role to join the tailnet,
- Creates a new user account with sudo privileges (specified by `created_username` variable) using `vault_infrastructure.tailscale_key` from the vault.
- Configures passwordless sudo for the sudo group
- Sets up SSH key authentication using your local `~/.ssh/id_ed25519.pub` key
- Disables root password authentication
### System Packages ## Requirements
- Installs `aptitude` for better package management - `ansible-galaxy install -r requirements.yml` (for `artis3n.tailscale`).
- Installs essential packages: - `~/.ssh/id_ed25519.pub` on the machine running Ansible.
- `curl` - HTTP client - Root (or another sudo user) SSH access to the new server.
- `vim` - Text editor
- `git` - Version control
- `ufw` - Uncomplicated Firewall
### Security Configuration
- Configures UFW firewall to:
- Allow SSH connections
- Enable firewall with default deny policy
- Hardens SSH configuration
## Variables Required
- `created_username`: The username to create (typically set in bootstrap.yml)
- `tailscale_key`: Tailscale authentication key (prompted during playbook run)
## Dependencies
- Requires the `artis3n.tailscale` role for VPN setup
- Requires your SSH public key at `~/.ssh/id_ed25519.pub`
## Usage ## Usage
```bash ```bash
ansible-playbook bootstrap.yml -i hosts.yml .venv/bin/ansible-playbook bootstrap.yml -i hosts.yml --vault-password-file vault_pass
``` ```
This role is designed to be run once on a fresh server before deploying other services. Once the server is on Tailscale, put its Tailscale IP in `hosts.yml` under
`docker` and use `bin/deploy` from then on.
roles/common/README.md
+8 -18
View File
@@ -1,23 +1,13 @@
# Common Role # Common Role
## Purpose Base setup applied first by `site.yml` (tag `common`).
Provides shared configuration and security setup that applies to all servers in the infrastructure.
## What It Does ## What it does
- Installs `aptitude` and `jq`.
### System Packages - Makes sure UFW is enabled with default deny. The SSH rule comes from the
- Installs `aptitude` for better package management and dependency resolution bootstrap role, and container rules from `ufw-docker` in the docker role.
- Updates package cache to ensure latest package information
### Security Configuration
- Enables UFW (Uncomplicated Firewall) with default deny policy
- Provides baseline firewall protection for all managed servers
## Usage ## Usage
This role is automatically applied to all servers in the infrastructure as part of the main site.yml playbook. It ensures consistent base configuration across all managed systems. ```bash
bin/deploy --tags common
## Dependencies ```
None - this is a foundational role that other roles can depend on.
## Notes
This role is designed to be lightweight and provide only the most essential common functionality. Server-specific configurations should be handled by dedicated roles like `docker` or `bootstrap`.
roles/cron/README.md
+27 -26
View File
@@ -1,35 +1,36 @@
# Cron Role # Cron Role
## Purpose Cron jobs on the server (tag `cron`). "Cron jobs" in this repo always means
Manages scheduled tasks and automated maintenance jobs for the infrastructure. jobs on the server, never on the machine running Ansible.
## What It Does ## Jobs
### Warhammer RSS Feed Updater | Time | User | Job |
- Copies `update_warhammer_feed.sh` script to `/usr/local/bin/` with executable permissions |---|---|---|
- Creates a daily cron job that runs at 09:10 AM | 08:00 daily | phil | `~/.local/bin/run_dockcheck.sh`: container updates |
- The script performs these actions: | 09:10 daily | root | `/usr/local/bin/update_warhammer_feed.sh`: Warhammer RSS feed |
1. Creates a temporary directory `/tmp/warhammer_feed`
2. Runs a custom Docker container (`git.thesatelliteoflove.com/phil/rss-warhammer`) to generate RSS feed
3. Copies the generated `warhammer_rss_feed.xml` to `/opt/stacks/caddy/site/tsol/feeds/`
## Files Managed ### Dockcheck (container updates)
- `/usr/local/bin/update_warhammer_feed.sh` - RSS feed update script - Downloads [dockcheck](https://github.com/mag37/dockcheck) and its Gotify
- Cron job: "Update Warhammer RSS Feed" (daily at 09:10) notify templates into `/home/phil/.local/bin/`.
- Templates `/home/phil/.config/dockcheck.config` from `dockcheck.config.j2`
## Dependencies (Gotify token from `vault_dockcheck`).
- Requires Docker to be installed and running - Settings: `OnlyLabel=true` and `AutoMode=true`, so containers labelled
- Depends on the following Docker stacks being deployed: `mag37.dockcheck.update=true` are pulled and recreated automatically; any
- Custom RSS generator container at `git.thesatelliteoflove.com/phil/rss-warhammer` other container with an update gets a Gotify notice. `dawarich_db` and
- Caddy web server stack at `/opt/stacks/caddy/` `dawarich_redis` are skipped.
- `run_dockcheck.sh` logs to `/var/log/dockcheck/dockcheck.log`.
## Usage
This role is automatically applied as part of the main site.yml playbook with the `cron` tag.
To see what needs updating by hand, run dockcheck on the server:
```bash ```bash
# Deploy only cron jobs ssh phil@100.70.169.99 '~/.local/bin/dockcheck.sh -n'
ansible-playbook site.yml -i hosts.yml --tags cron
``` ```
## Customization ### Warhammer RSS feed
To add additional cron jobs, create new tasks in the main.yml file following the same pattern as the Warhammer feed updater. `update_warhammer_feed.sh` runs the `git.thesatelliteoflove.com/phil/rss-warhammer`
container to build `warhammer_rss_feed.xml` and copies it to
`/opt/stacks/caddy/site/tsol/feeds/`, where Caddy serves it on the main site.
## Adding a job
Add a `cron` task to `tasks/main.yml` (copy the script with `copy` first if it
needs one), then `bin/deploy --tags cron`.
roles/docker/README.md
+120 -197
View File
@@ -1,221 +1,144 @@
# Docker Role # Docker Role
## Purpose Installs Docker and deploys all 24 services. Each service is a Docker Compose
Deploys and manages a comprehensive self-hosted infrastructure with 24 containerized services organized into logical categories, transforming a server into a personal cloud platform with authentication, media management, productivity tools, and development services. stack in `/opt/stacks/<service>/` (Dockge lives in `/opt/dockge/`), built from
Jinja2 templates in this role.
## Architecture Overview ## What runs before the services
### Network Configuration `tasks/main.yml` does the system setup, then imports each category:
- **External Network**: All services connect to shared Docker network (configurable)
- **Reverse Proxy**: Caddy handles all ingress traffic with automatic HTTPS
- **Service Discovery**: Container-to-container communication using service names
- **Firewall Integration**: UFW-Docker script properly configures firewall rules
### Security Features 1. Base packages (`python3-pip`, `virtualenv`, `duf`, …).
- **Centralized SSO**: Pocket ID provides passkey OIDC authentication for most services 2. Shell setup (`shell.yml`): zsh as the login shell, Atuin (installed and
- **Network Isolation**: Services restricted to appropriate network segments logged in once, dotfile sync on), and `bat`.
- **Container Hardening**: Non-root users, capability dropping, security options 3. Docker CE from Docker's apt repository.
- **Secret Management**: Ansible vault for sensitive configuration 4. `ufw-docker.sh` copied to `/usr/local/bin/` and its rules installed (UFW is
- **Variable Management**: Centralized variable hierarchy using group_vars structure restarted once after install).
5. The shared `lava` network (see below).
6. `/opt/stacks` and `/opt/dockge`.
## Services Deployed (Organized by Category) ## Services by category
### Infrastructure (`infrastructure/`) Tags: each category file has a category tag, each service its own tag.
- **Caddy** - Reverse proxy with automatic HTTPS (static IP: 172.20.0.5)
- **Pocket ID** - Passkey OIDC provider
- **Dockge** - Docker compose stack management UI
### Development (`development/`) ### Infrastructure (`infrastructure`)
- **Gitea** - Self-hosted Git with CI/CD runners | Tag | Service | Notes |
- **Code Server** - VS Code in the browser |---|---|---|
- **ByteStash** - Code snippet management and organization | `caddy` | Caddy | Fixed IP 172.20.0.5. Ports 80/443 (+443/udp). `Caddyfile.j2` changes restart it via a handler; the stack itself is deployed with `recreate: never`. Static sites are served from `/opt/stacks/caddy/site/`. |
| `pocketid` | Pocket ID v2 | Passkey OIDC provider at `id.`. OIDC clients are created in its web UI; their IDs and secrets go in the vault. |
| `gondulf` | Gondulf | IndieAuth server for the main site (own image from Gitea). Sends login codes by email through Resend. |
| `dockge` | Dockge | Compose UI on port 5001, managing `/opt/stacks`. |
### Media (`media/`) ### Development (`development`)
- **Audiobookshelf** - Audiobook server (Pocket ID login) | Tag | Service | Notes |
- **Kavita** - E-book server (Pocket ID login) |---|---|---|
- **Ghost** - Modern blogging platform | `gitea` | Gitea + act_runner | Fixed IP 172.20.0.6 so the ufw-docker rule for SSH (host port 222) stays valid. Pocket ID login is set up in Gitea's admin UI; OpenID 2.0 login is off. |
- **Pinchflat** - YouTube video archiving | `bytestash` | ByteStash | Pocket ID through env vars. |
- **Pinry** - Pinterest-like image board
- **Karakeep** - Bookmark management with AI tagging
- **Manyfold** - 3D model file organization
### Productivity (`productivity/`) ### Media (`media`)
- **Paperless-ngx** - Document management with OCR | Tag | Service | Notes |
- **MMDL** - Task and calendar management with CalDAV integration |---|---|---|
- **Baikal** - CalDAV/CardDAV server | `audiobookshelf` | Audiobookshelf | Pocket ID set up in the app's settings. Podcasts are not used. |
- **Syncthing** - Decentralized file sync | `kavita` | Kavita | Ebooks at `books.`. Also handles OPDS and KOReader sync (replaced Calibre-Web, dir2opds and kosync). Pocket ID set up in the app's settings. |
- **Heyform** - Form builder and surveys | `karakeep`, `hoarder` | Karakeep | Files are still named `hoarder*`. Web, headless Chrome and Meilisearch. Pocket ID through `hoarder-env.j2`. |
- **Dawarich** - Location tracking | `manyfold` | Manyfold | Pocket ID through env vars. |
- **Palmr** - File sharing service
- **Obsidian LiveSync** - CouchDB backend for note synchronization
### Communication (`communication/`) ### Productivity (`productivity`)
- **GoToSocial** - Lightweight ActivityPub server | Tag | Service | Notes |
- **Postiz** - Social media management |---|---|---|
| `paperlessngx` | Paperless-ngx | Redis, Gotenberg, Tika. Pocket ID through django-allauth (`paperlessngx.env.j2`). Daily S3 backup of the data volume. |
| `baikal` | Baikal | CalDAV/CardDAV. Caddy redirects the `.well-known` paths. |
| `syncthing` | Syncthing | Ports on the Tailscale IP only. |
| `heyform` | HeyForm | MongoDB (Percona) and KeyDB. On `forms.nerder.land`. |
| `dawarich` | Dawarich | App, Sidekiq, PostGIS, Redis. Based on the upstream production compose file. |
| `obsidian-livesync` | CouchDB for Obsidian LiveSync | Port 5984 on the Tailscale IP only. Has a restart handler. |
| `gokapi` | Gokapi | Pocket ID login. `config.json` is only written on a fresh install, because Gokapi rewrites it on upgrade. To change it, delete `/opt/stacks/gokapi/config/config.json` and redeploy. |
### Monitoring (`monitoring/`) ### Communication (`communication`)
- **Change Detection** - Website monitoring | Tag | Service | Notes |
- **Apprise API** - Unified notifications |---|---|---|
- **Gotify** - Self-hosted push notification service | `gotosocial` | GoToSocial | Pocket ID through env vars. Account domain is the apex; Caddy serves WebFinger there. Daily S3 backup of its volume. |
| `listmonk` | Listmonk | Postgres 17. On `listmonk.nerder.land`. |
| `starpunk` | StarPunk | Micropub server for the main site (own image from Gitea). Logs in through Gondulf. |
## Deployment Patterns ### Monitoring (`monitoring`)
| Tag | Service | Notes |
|---|---|---|
| `changedetection` | Changedetection | With a sockpuppetbrowser for JavaScript pages. |
| `appriseapi` | Apprise API | Port 8000 on the Tailscale IP only. Other containers reach it at `http://apprise:8000`. |
| `gotify` | Gotify + iGotify assistant | Push notifications. Dockcheck sends its update notices here. |
| `cronmaster` | Cronmaster | Cron UI, port 40123 on the Tailscale IP only. |
### Standardized Service Deployment ## Networking
Each service follows a consistent pattern:
1. Creates `/opt/stacks/[service-name]` directory structure
2. Generates Docker Compose file from Jinja2 template
3. Deploys using `community.docker.docker_compose_v2`
4. Configures environment variables from vault secrets
### Template System ### Shared network (`lava`)
- **Compose Templates**: `.j2` files in `templates/` for dynamic configuration Every stack joins the external network `lava` (`172.20.0.0/16`). Docker only
- **Environment Templates**: Separate `.env.j2` files for services requiring environment variables hands out dynamic addresses from `172.20.128.0/17`, so `172.20.0.x` is free
- **Variable Substitution**: Uses centralized variable hierarchy from group_vars structure for fixed IPs: Caddy (.5) and Gitea (.6). Both need fixed IPs because the
- **Domain Management**: Centralized domain and subdomain configuration ufw-docker rules point at container IPs. The role creates the network if it
- **Network Configuration**: Standardized Docker network and IP address management is missing and warns if it lacks the reserved range; `migrate-lava-network.yml`
fixes an old network.
## Shell Environment Setup ### Hairpinning
The role also configures the shell environment: Containers can't reach the server's own public IP, so any app that calls
- **Zsh Installation**: Sets zsh as default shell another of our domains (usually Pocket ID for OIDC) maps it to Caddy:
- **Atuin**: Command history sync and search
- **Bat**: Enhanced `cat` command with syntax highlighting ```yaml
extra_hosts:
- "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}"
```
### Firewall
`ufw-docker.sh` makes UFW apply to published container ports. Only Caddy
(80/tcp, 443/tcp, 443/udp) and Gitea (22/tcp, published as 222) are allowed
in from the internet; the Caddy and Gitea task files add those rules.
## Updates
Containers labelled `mag37.dockcheck.update: true` are updated automatically
each morning by dockcheck (set up in the cron role). Containers without the
label (Gondulf, StarPunk, Listmonk, HeyForm, Dockge and most databases) only
get a Gotify notice. Dawarich's database and Redis are skipped entirely.
Gondulf and StarPunk use pinned version tags.
## Adding a service
1. `templates/<service>-compose.yml.j2` (plus `-env.j2` if needed). Join the
external `{{ docker.network_name }}` network and add `extra_hosts` for any
of our domains the app calls.
2. `tasks/<category>/<service>.yml`: make `/opt/stacks/<service>`, template
the files, deploy with `community.docker.docker_compose_v2`. Use
`diff: false` or `no_log: true` on tasks that write secrets.
3. Import it in `tasks/<category>/main.yml` with a tag named after the service.
4. Subdomain in `group_vars/all/domains.yml`, route in `templates/Caddyfile.j2`,
A record in `dns.yml`.
5. Secrets under `vault_<service>` with `bin/vault-run`.
## Files
## File Organization
``` ```
roles/docker/ roles/docker/
├── tasks/ ├── tasks/
│ ├── main.yml # Orchestrates all deployments │ ├── main.yml # System setup, network, category imports
│ ├── shell.yml # Shell environment setup │ ├── shell.yml # zsh, Atuin, bat
│ ├── infrastructure/ │ ├── infrastructure/ # caddy, pocketid, gondulf (+ dockge in main.yml)
│ │ ├── main.yml # Infrastructure category orchestrator │ ├── development/ # gitea, bytestash
│ │ ├── caddy.yml # Reverse proxy │ ├── media/ # audiobookshelf, kavita, hoarder (Karakeep), manyfold
│ │ └── pocketid.yml # Authentication │ ├── productivity/ # paperlessngx, baikal, syncthing, heyform, dawarich, obsidian-livesync, gokapi
│ ├── development/ │ ├── communication/ # gotosocial, listmonk, starpunk
│ │ ├── main.yml # Development category orchestrator │ └── monitoring/ # changedetection, appriseapi, gotify, cronmaster
│ │ ├── gitea.yml # Git hosting ├── templates/ # <service>-compose.yml.j2, env and config templates, Caddyfile.j2
│ │ └── bytestash.yml # Code snippets ├── files/ufw-docker.sh # UFW integration for Docker
│ ├── media/ # Media services (7 services) └── handlers/main.yml # restart caddy, restart obsidian-livesync
│ ├── productivity/ # Productivity services (7 services)
│ ├── communication/ # Communication services (2 services)
│ └── monitoring/ # Monitoring services (3 services)
├── templates/
│ ├── [service]-compose.yml.j2 # Docker Compose templates (all templated)
│ ├── [service]-env.j2 # Environment variable templates
│ └── [service]-*.j2 # Service-specific templates
├── files/
│ ├── Caddyfile # Caddy configuration
│ ├── ufw-docker.sh # Firewall integration script
│ ├── client # Matrix well-known client file
│ └── server # Matrix well-known server file
└── handlers/
└── main.yml # Service restart handlers
``` ```
## Usage
### Deploy All Services
```bash
ansible-playbook site.yml -i hosts.yml --tags docker
```
### Deploy by Service Category
```bash
# Deploy entire service categories
ansible-playbook site.yml -i hosts.yml --tags infrastructure
ansible-playbook site.yml -i hosts.yml --tags development
ansible-playbook site.yml -i hosts.yml --tags media
ansible-playbook site.yml -i hosts.yml --tags productivity
ansible-playbook site.yml -i hosts.yml --tags communication
ansible-playbook site.yml -i hosts.yml --tags monitoring
# Deploy multiple categories
ansible-playbook site.yml -i hosts.yml --tags infrastructure,monitoring
```
### Deploy Individual Services
```bash
# Deploy specific services
ansible-playbook site.yml -i hosts.yml --tags pocketid
ansible-playbook site.yml -i hosts.yml --tags gitea,bytestash
ansible-playbook site.yml -i hosts.yml --tags paperlessngx
```
## Service-Specific Notes
### MMDL (Task Management)
- **URL**: https://tasks.thesatelliteoflove.com
- **Initial Setup**: Visit `/install` endpoint first to run database migrations
- **Authentication**: Integrates with the Pocket ID OIDC provider
- **Database**: Uses MySQL 8.0 with automatic schema migration
- **Features**: CalDAV integration, multiple account support, task management
## Dependencies
### System Requirements
- Docker CE installed and running
- UFW firewall configured
- DNS records pointing to the server
- Valid SSL certificates (handled automatically by Caddy)
### External Services
- **DNS**: Requires subdomains configured for each service
- **Email**: Gitea uses Resend for notifications
- **Storage**: All services persist data to `/opt/stacks/[service]/`
## Configuration
### Variable Structure
The role uses a centralized variable hierarchy in `group_vars/all/`:
- **domains.yml**: Domain and subdomain mappings for all services
- **infrastructure.yml**: Network configuration, Docker settings, and system parameters
- **vault.yml**: Encrypted secrets including API keys, passwords, and OAuth credentials
- **services.yml**: Service-specific configuration and feature flags
### Required Variables (in vault.yml)
- Authentication credentials for various services (vault_*)
- API keys for external integrations
- OAuth client secrets for SSO integration
- Database passwords and connection strings
- SMTP credentials for notifications
### Network Configuration
Services expect to be accessible via subdomains of configured domains:
- `id.thesatelliteoflove.com` - Pocket ID
- `git.thesatelliteoflove.com` - Gitea
- `books.thesatelliteoflove.com` - Kavita
- `tasks.thesatelliteoflove.com` - MMDL
- (and many more...)
## Monitoring & Management
### Operational Features
- Automatic container restart policies
- Health checks for database services
- Centralized logging and monitoring
- Backup-ready data structure in `/opt/stacks/`
## Security Considerations
### Network Security
- UFW-Docker integration for proper firewall rules
- Services isolated to appropriate network segments
- Restricted access for sensitive tools (Stirling PDF)
### Authentication
- Centralized SSO through Pocket ID for most services
- OAuth integration where supported
- Secure secret management through Ansible vault
### Container Security
- Non-root container execution (UID/GID 1000:1000)
- Security options: `no-new-privileges: true`
- Capability dropping and minimal permissions
## Troubleshooting ## Troubleshooting
### Common Issues - **OIDC login fails with a connection error:** the container is missing the
- **Database Connection**: Ensure MySQL containers use proper authentication plugins `extra_hosts` entry for `id.thesatelliteoflove.com`.
- **OAuth Discovery**: Check issuer URLs don't have trailing slashes - **OIDC discovery fails:** check the issuer URL has no trailing slash.
- **Migration Failures**: Visit service `/install` endpoints for database setup - **Caddy fails with "Address already in use":** another container took
- **Network Issues**: Verify containers are on the same Docker network 172.20.0.5; the network lacks the reserved range. Run
`migrate-lava-network.yml`.
- **Gitea SSH stops working after a recreate:** Gitea lost its fixed IP;
redeploy with `--tags gitea` to restore it and the firewall rule.
todo.md
+45 -128
View File
@@ -1,141 +1,58 @@
# Infrastructure Improvements TODO # Infrastructure Improvements TODO
## High Priority (Quick Wins) Current state (2026-10-31): 24 services in 6 categories, all compose files
templated, Pocket ID as SSO (Authentik retired), multi-agent guard rails
(`bin/worktree`, `bin/deploy`, `bin/vault-run`, `deploy-guard.yml`).
### 1. Split the massive docker role ✅ COMPLETED ## Done
- **Current Issue**: `roles/docker/tasks/main.yml` has 20+ services in one file (176 lines) - Split the docker role into categories with category and service tags.
- **Solution**: Break into logical service groups: - Central variables in `group_vars/all/` (`domains.yml`, `infrastructure.yml`,
``` `vault.yml`).
roles/docker/tasks/ - Converted every static compose file to a template.
├── main.yml (orchestrator) - Removed unused services, most recently Authentik, Calibre-Web, dir2opds,
├── infrastructure/ (caddy, pocketid, dockge) kosync, Sneaky Klaus and Audiobookshelf podcasts.
├── development/ (gitea, codeserver, bytestash) - Reserved address range on the `lava` network so fixed IPs can't be taken.
├── media/ (audiobookshelf, kavita, ghost, pinchflat, pinry, hoarder, manyfold) - Automatic container updates with dockcheck for labelled containers.
├── productivity/ (paperless, baikal, syncthing, mmdl, heyform, dawarich, palmr, obsidian-livesync)
├── communication/ (gotosocial, postiz)
└── monitoring/ (changedetection, appriseapi, gotify)
```
- **COMPLETED**: All services organized into logical categories with category-level tags
### 2. Standardize variable management ✅ COMPLETED ## Open
- **Current Issue**: Secrets in single encrypted file, no clear variable hierarchy
- **Solution**: Create proper variable structure:
```
group_vars/
├── all/
│ ├── domains.yml (domain and subdomain mappings)
│ ├── infrastructure.yml (network config, Docker settings)
│ ├── vault.yml (encrypted secrets with vault_ prefix)
│ └── services.yml (service configuration flags)
```
- **COMPLETED**: Implemented comprehensive variable hierarchy, updated all templates to use centralized variables, fixed service tag isolation
### 3. Template consolidation ✅ PARTIALLY COMPLETED ### Reusable template patterns
- **Current Issue**: Many compose templates repeat patterns, some services used static files Many compose templates repeat the same network, restart and label blocks.
- **Solution**: Create reusable template includes with standard service template structure Pull the shared parts into includes or macros.
- **COMPLETED**: Converted all static compose files (caddy, dockge, hoarder) to Jinja2 templates
- **REMAINING**: Create reusable template patterns for common configurations
## Security & Reliability ### Health checks
Only Pocket ID, Gondulf, Dawarich, Listmonk and Syncthing define health checks
in their templates. Add them to the rest where the image supports it.
### 4. Add health checks ### Backups
- **Issue**: Most services lack proper healthcheck configurations in compose templates Only Paperless-ngx and GoToSocial back up (daily to S3 with
- **Solution**: Implement comprehensive health monitoring with standardized healthcheck patterns `offen/docker-volume-backup`, 7-day retention). Still to do:
- Database dumps for the Postgres/MongoDB/CouchDB services.
- Volume backups for the file-based services (Kavita, Audiobookshelf, Gitea,
Pocket ID, …).
- A tested restore.
### 5. Implement backup strategy ### Network segmentation
- **Issue**: No automated backups for 25 services and their data Every service shares the `lava` network. Consider separate frontend, backend
- **Solution**: Add backup role with: and database networks.
- Database dumps for PostgreSQL services
- Volume backups for file-based services
- Rotation policies
- Restoration testing
### 6. Network segmentation ### Security hardening
- **Issue**: All services share one Docker network - Add `no-new-privileges` and dropped capabilities where images allow it
- **Solution**: Separate into: (only Manyfold has them now). Cronmaster runs as root.
- `frontend` (Public-facing services) - fail2ban (or Caddy rate limits) in front of login pages.
- `backend` (Internal services only)
- `database` (Database access only)
### 7. Security hardening ### CI with Gitea Actions
- Remove unnecessary `user: root` from services Syntax check and `--check` runs on push, using the existing Gitea runner.
- Add security contexts to all containers
- Implement least-privilege access patterns
- Add fail2ban for authentication services
## Automation Opportunities ### Configuration drift detection
Catch manual changes on the server, e.g. a scheduled `bin/deploy --check`.
### 8. CI/CD with Gitea Actions ### Service start order
- Leverage self-hosted Gitea for: Apps that use Pocket ID don't wait for it to be up.
- Ansible syntax validation
- Service configuration testing
- Automated deployment triggers
- Rollback capabilities
### 9. Configuration drift detection ### Ansible clean-up
- Add validation tasks to catch manual changes - Replace the deprecated `apt_key` task for Docker's repo (it also fetches the
- Implement configuration validation with proper assertions Ubuntu key for a Debian repo).
- Use `ansible.builtin.` names everywhere.
### 10. Service dependency management - The Dockge compose file is templated twice (`tasks/main.yml` and
- **Issue**: Some services depend on Pocket ID SSO but no startup ordering `tasks/infrastructure/main.yml`).
- **Solution**: Implement dependency checking and startup ordering
### 11. Ansible best practices
- Replace deprecated `apt_key` with proper patterns
- Use `ansible.builtin` FQCN consistently
- Add `check_mode` support
- Implement proper idempotency checks
### 12. Documentation automation
- Auto-generate service inventory
- Create service documentation templates
- Implement automated documentation updates
## Implementation Roadmap
### Week 1: Foundation
- [x] Document improvements in todo.md
- [x] Reorganize docker role structure
- [x] Convert static compose files to templates
- [x] Remove unused services (beaver, grist, stirlingpdf, tasksmd, redlib)
- [x] Clean up templates and files directories
- [x] Implement variable hierarchy
- [ ] Create reusable template patterns
### Week 2: Security & Monitoring
- [ ] Add health checks
- [ ] Implement backup strategy
- [ ] Security hardening
### Week 3: Automation
- [ ] CI/CD pipeline setup
- [ ] Configuration validation
- [ ] Documentation automation
### Week 4: Advanced Features
- [ ] Network segmentation
- [ ] Dependency management
- [ ] Monitoring dashboard
## Completed Work Summary
### ✅ Major Accomplishments
- **Docker Role Reorganization**: Split monolithic 176-line main.yml into 6 logical service categories
- **Template Standardization**: Converted all static compose files to Jinja2 templates
- **Service Cleanup**: Removed 5 unused/broken services (beaver, grist, stirlingpdf, tasksmd, redlib)
- **Category-Based Deployment**: Can now deploy services by category using tags (infrastructure, media, etc.)
- **Variable Management**: Implemented comprehensive centralized variable hierarchy with proper secret organization
- **Service Tag Isolation**: Fixed service tags to deploy individual services only (not entire categories)
- **Documentation Updates**: Updated all README files and CLAUDE.md to reflect new architecture
### 📊 Current Stats
- **25 active services** organized into 6 categories
- **100% templated** compose files (no static files)
- **6 service directories** for logical organization
- **Clean file structure** with only essential static files
## Notes
- Current architecture is solid and much better organized for long-term maintainability
- Focus on high-impact, low-effort improvements first
- Leverage existing infrastructure (Gitea, Pocket ID) for automation
- Template-driven approach enables future dynamic configuration