phil/ansible
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Bring the docs up to date with the running stack

Browse Source 
The READMEs, CLAUDE.md and todo.md still described Authentik, services
removed long ago (Ghost, MMDL, Postiz, Pinchflat...) and two servers. Now
they list the 24 services actually deployed (checked against docker ps),
explain that netcup and docker-01 are one machine, document Gondulf,
StarPunk, Listmonk, Gokapi and dockcheck, and use bin/deploy throughout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Phil
2026-10-31 11:30:50 -06:00
co-authored by Claude Opus 5.5
parent 0282debf07
commit 0c14ee5b1d
8 changed files with 426 additions and 639 deletions
Download Patch File Download Diff File Expand all files Collapse all files
roles/docker/README.md
+120 -197
View File
@@ -1,221 +1,144 @@
# Docker Role
## Purpose
Deploys and manages a comprehensive self-hosted infrastructure with 24 containerized services organized into logical categories, transforming a server into a personal cloud platform with authentication, media management, productivity tools, and development services.
Installs Docker and deploys all 24 services. Each service is a Docker Compose
stack in `/opt/stacks/<service>/` (Dockge lives in `/opt/dockge/`), built from
Jinja2 templates in this role.
## Architecture Overview
## What runs before the services
### Network Configuration
- **External Network**: All services connect to shared Docker network (configurable)
- **Reverse Proxy**: Caddy handles all ingress traffic with automatic HTTPS
- **Service Discovery**: Container-to-container communication using service names
- **Firewall Integration**: UFW-Docker script properly configures firewall rules
`tasks/main.yml` does the system setup, then imports each category:
### Security Features
- **Centralized SSO**: Pocket ID provides passkey OIDC authentication for most services
- **Network Isolation**: Services restricted to appropriate network segments
- **Container Hardening**: Non-root users, capability dropping, security options
- **Secret Management**: Ansible vault for sensitive configuration
- **Variable Management**: Centralized variable hierarchy using group_vars structure
1. Base packages (`python3-pip`, `virtualenv`, `duf`, …).
2. Shell setup (`shell.yml`): zsh as the login shell, Atuin (installed and
logged in once, dotfile sync on), and `bat`.
3. Docker CE from Docker's apt repository.
4. `ufw-docker.sh` copied to `/usr/local/bin/` and its rules installed (UFW is
restarted once after install).
5. The shared `lava` network (see below).
6. `/opt/stacks` and `/opt/dockge`.
## Services Deployed (Organized by Category)
## Services by category
### Infrastructure (`infrastructure/`)
- **Caddy** - Reverse proxy with automatic HTTPS (static IP: 172.20.0.5)
- **Pocket ID** - Passkey OIDC provider
- **Dockge** - Docker compose stack management UI
Tags: each category file has a category tag, each service its own tag.
### Development (`development/`)
- **Gitea** - Self-hosted Git with CI/CD runners
- **Code Server** - VS Code in the browser
- **ByteStash** - Code snippet management and organization
### Infrastructure (`infrastructure`)
| Tag | Service | Notes |
|---|---|---|
| `caddy` | Caddy | Fixed IP 172.20.0.5. Ports 80/443 (+443/udp). `Caddyfile.j2` changes restart it via a handler; the stack itself is deployed with `recreate: never`. Static sites are served from `/opt/stacks/caddy/site/`. |
| `pocketid` | Pocket ID v2 | Passkey OIDC provider at `id.`. OIDC clients are created in its web UI; their IDs and secrets go in the vault. |
| `gondulf` | Gondulf | IndieAuth server for the main site (own image from Gitea). Sends login codes by email through Resend. |
| `dockge` | Dockge | Compose UI on port 5001, managing `/opt/stacks`. |
### Media (`media/`)
- **Audiobookshelf** - Audiobook server (Pocket ID login)
- **Kavita** - E-book server (Pocket ID login)
- **Ghost** - Modern blogging platform
- **Pinchflat** - YouTube video archiving
- **Pinry** - Pinterest-like image board
- **Karakeep** - Bookmark management with AI tagging
- **Manyfold** - 3D model file organization
### Development (`development`)
| Tag | Service | Notes |
|---|---|---|
| `gitea` | Gitea + act_runner | Fixed IP 172.20.0.6 so the ufw-docker rule for SSH (host port 222) stays valid. Pocket ID login is set up in Gitea's admin UI; OpenID 2.0 login is off. |
| `bytestash` | ByteStash | Pocket ID through env vars. |
### Productivity (`productivity/`)
- **Paperless-ngx** - Document management with OCR
- **MMDL** - Task and calendar management with CalDAV integration
- **Baikal** - CalDAV/CardDAV server
- **Syncthing** - Decentralized file sync
- **Heyform** - Form builder and surveys
- **Dawarich** - Location tracking
- **Palmr** - File sharing service
- **Obsidian LiveSync** - CouchDB backend for note synchronization
### Media (`media`)
| Tag | Service | Notes |
|---|---|---|
| `audiobookshelf` | Audiobookshelf | Pocket ID set up in the app's settings. Podcasts are not used. |
| `kavita` | Kavita | Ebooks at `books.`. Also handles OPDS and KOReader sync (replaced Calibre-Web, dir2opds and kosync). Pocket ID set up in the app's settings. |
| `karakeep`, `hoarder` | Karakeep | Files are still named `hoarder*`. Web, headless Chrome and Meilisearch. Pocket ID through `hoarder-env.j2`. |
| `manyfold` | Manyfold | Pocket ID through env vars. |
### Communication (`communication/`)
- **GoToSocial** - Lightweight ActivityPub server
- **Postiz** - Social media management
### Productivity (`productivity`)
| Tag | Service | Notes |
|---|---|---|
| `paperlessngx` | Paperless-ngx | Redis, Gotenberg, Tika. Pocket ID through django-allauth (`paperlessngx.env.j2`). Daily S3 backup of the data volume. |
| `baikal` | Baikal | CalDAV/CardDAV. Caddy redirects the `.well-known` paths. |
| `syncthing` | Syncthing | Ports on the Tailscale IP only. |
| `heyform` | HeyForm | MongoDB (Percona) and KeyDB. On `forms.nerder.land`. |
| `dawarich` | Dawarich | App, Sidekiq, PostGIS, Redis. Based on the upstream production compose file. |
| `obsidian-livesync` | CouchDB for Obsidian LiveSync | Port 5984 on the Tailscale IP only. Has a restart handler. |
| `gokapi` | Gokapi | Pocket ID login. `config.json` is only written on a fresh install, because Gokapi rewrites it on upgrade. To change it, delete `/opt/stacks/gokapi/config/config.json` and redeploy. |
### Monitoring (`monitoring/`)
- **Change Detection** - Website monitoring
- **Apprise API** - Unified notifications
- **Gotify** - Self-hosted push notification service
### Communication (`communication`)
| Tag | Service | Notes |
|---|---|---|
| `gotosocial` | GoToSocial | Pocket ID through env vars. Account domain is the apex; Caddy serves WebFinger there. Daily S3 backup of its volume. |
| `listmonk` | Listmonk | Postgres 17. On `listmonk.nerder.land`. |
| `starpunk` | StarPunk | Micropub server for the main site (own image from Gitea). Logs in through Gondulf. |
## Deployment Patterns
### Monitoring (`monitoring`)
| Tag | Service | Notes |
|---|---|---|
| `changedetection` | Changedetection | With a sockpuppetbrowser for JavaScript pages. |
| `appriseapi` | Apprise API | Port 8000 on the Tailscale IP only. Other containers reach it at `http://apprise:8000`. |
| `gotify` | Gotify + iGotify assistant | Push notifications. Dockcheck sends its update notices here. |
| `cronmaster` | Cronmaster | Cron UI, port 40123 on the Tailscale IP only. |
### Standardized Service Deployment
Each service follows a consistent pattern:
1. Creates `/opt/stacks/[service-name]` directory structure
2. Generates Docker Compose file from Jinja2 template
3. Deploys using `community.docker.docker_compose_v2`
4. Configures environment variables from vault secrets
## Networking
### Template System
- **Compose Templates**: `.j2` files in `templates/` for dynamic configuration
- **Environment Templates**: Separate `.env.j2` files for services requiring environment variables
- **Variable Substitution**: Uses centralized variable hierarchy from group_vars structure
- **Domain Management**: Centralized domain and subdomain configuration
- **Network Configuration**: Standardized Docker network and IP address management
### Shared network (`lava`)
Every stack joins the external network `lava` (`172.20.0.0/16`). Docker only
hands out dynamic addresses from `172.20.128.0/17`, so `172.20.0.x` is free
for fixed IPs: Caddy (.5) and Gitea (.6). Both need fixed IPs because the
ufw-docker rules point at container IPs. The role creates the network if it
is missing and warns if it lacks the reserved range; `migrate-lava-network.yml`
fixes an old network.
## Shell Environment Setup
The role also configures the shell environment:
- **Zsh Installation**: Sets zsh as default shell
- **Atuin**: Command history sync and search
- **Bat**: Enhanced `cat` command with syntax highlighting
### Hairpinning
Containers can't reach the server's own public IP, so any app that calls
another of our domains (usually Pocket ID for OIDC) maps it to Caddy:
```yaml
extra_hosts:
- "{{ subdomains.pocketid }}:{{ docker.hairpin_ip }}"
```
### Firewall
`ufw-docker.sh` makes UFW apply to published container ports. Only Caddy
(80/tcp, 443/tcp, 443/udp) and Gitea (22/tcp, published as 222) are allowed
in from the internet; the Caddy and Gitea task files add those rules.
## Updates
Containers labelled `mag37.dockcheck.update: true` are updated automatically
each morning by dockcheck (set up in the cron role). Containers without the
label (Gondulf, StarPunk, Listmonk, HeyForm, Dockge and most databases) only
get a Gotify notice. Dawarich's database and Redis are skipped entirely.
Gondulf and StarPunk use pinned version tags.
## Adding a service
1. `templates/<service>-compose.yml.j2` (plus `-env.j2` if needed). Join the
external `{{ docker.network_name }}` network and add `extra_hosts` for any
of our domains the app calls.
2. `tasks/<category>/<service>.yml`: make `/opt/stacks/<service>`, template
the files, deploy with `community.docker.docker_compose_v2`. Use
`diff: false` or `no_log: true` on tasks that write secrets.
3. Import it in `tasks/<category>/main.yml` with a tag named after the service.
4. Subdomain in `group_vars/all/domains.yml`, route in `templates/Caddyfile.j2`,
A record in `dns.yml`.
5. Secrets under `vault_<service>` with `bin/vault-run`.
## Files
## File Organization
```
roles/docker/
├── tasks/
│ ├── main.yml # Orchestrates all deployments
│ ├── shell.yml # Shell environment setup
│ ├── infrastructure/
│ │ ├── main.yml # Infrastructure category orchestrator
│ │ ├── caddy.yml # Reverse proxy
│ │ └── pocketid.yml # Authentication
│ ├── development/
│ │ ├── main.yml # Development category orchestrator
│ │ ├── gitea.yml # Git hosting
│ │ └── bytestash.yml # Code snippets
│ ├── media/ # Media services (7 services)
│ ├── productivity/ # Productivity services (7 services)
│ ├── communication/ # Communication services (2 services)
│ └── monitoring/ # Monitoring services (3 services)
├── templates/
│ ├── [service]-compose.yml.j2 # Docker Compose templates (all templated)
│ ├── [service]-env.j2 # Environment variable templates
│ └── [service]-*.j2 # Service-specific templates
├── files/
│ ├── Caddyfile # Caddy configuration
│ ├── ufw-docker.sh # Firewall integration script
│ ├── client # Matrix well-known client file
│ └── server # Matrix well-known server file
└── handlers/
└── main.yml # Service restart handlers
│ ├── main.yml # System setup, network, category imports
│ ├── shell.yml # zsh, Atuin, bat
│ ├── infrastructure/ # caddy, pocketid, gondulf (+ dockge in main.yml)
│ ├── development/ # gitea, bytestash
│ ├── media/ # audiobookshelf, kavita, hoarder (Karakeep), manyfold
│ ├── productivity/ # paperlessngx, baikal, syncthing, heyform, dawarich, obsidian-livesync, gokapi
│ ├── communication/ # gotosocial, listmonk, starpunk
│ └── monitoring/ # changedetection, appriseapi, gotify, cronmaster
├── templates/ # <service>-compose.yml.j2, env and config templates, Caddyfile.j2
├── files/ufw-docker.sh # UFW integration for Docker
└── handlers/main.yml # restart caddy, restart obsidian-livesync
```
## Usage
### Deploy All Services
```bash
ansible-playbook site.yml -i hosts.yml --tags docker
```
### Deploy by Service Category
```bash
# Deploy entire service categories
ansible-playbook site.yml -i hosts.yml --tags infrastructure
ansible-playbook site.yml -i hosts.yml --tags development
ansible-playbook site.yml -i hosts.yml --tags media
ansible-playbook site.yml -i hosts.yml --tags productivity
ansible-playbook site.yml -i hosts.yml --tags communication
ansible-playbook site.yml -i hosts.yml --tags monitoring
# Deploy multiple categories
ansible-playbook site.yml -i hosts.yml --tags infrastructure,monitoring
```
### Deploy Individual Services
```bash
# Deploy specific services
ansible-playbook site.yml -i hosts.yml --tags pocketid
ansible-playbook site.yml -i hosts.yml --tags gitea,bytestash
ansible-playbook site.yml -i hosts.yml --tags paperlessngx
```
## Service-Specific Notes
### MMDL (Task Management)
- **URL**: https://tasks.thesatelliteoflove.com
- **Initial Setup**: Visit `/install` endpoint first to run database migrations
- **Authentication**: Integrates with the Pocket ID OIDC provider
- **Database**: Uses MySQL 8.0 with automatic schema migration
- **Features**: CalDAV integration, multiple account support, task management
## Dependencies
### System Requirements
- Docker CE installed and running
- UFW firewall configured
- DNS records pointing to the server
- Valid SSL certificates (handled automatically by Caddy)
### External Services
- **DNS**: Requires subdomains configured for each service
- **Email**: Gitea uses Resend for notifications
- **Storage**: All services persist data to `/opt/stacks/[service]/`
## Configuration
### Variable Structure
The role uses a centralized variable hierarchy in `group_vars/all/`:
- **domains.yml**: Domain and subdomain mappings for all services
- **infrastructure.yml**: Network configuration, Docker settings, and system parameters
- **vault.yml**: Encrypted secrets including API keys, passwords, and OAuth credentials
- **services.yml**: Service-specific configuration and feature flags
### Required Variables (in vault.yml)
- Authentication credentials for various services (vault_*)
- API keys for external integrations
- OAuth client secrets for SSO integration
- Database passwords and connection strings
- SMTP credentials for notifications
### Network Configuration
Services expect to be accessible via subdomains of configured domains:
- `id.thesatelliteoflove.com` - Pocket ID
- `git.thesatelliteoflove.com` - Gitea
- `books.thesatelliteoflove.com` - Kavita
- `tasks.thesatelliteoflove.com` - MMDL
- (and many more...)
## Monitoring & Management
### Operational Features
- Automatic container restart policies
- Health checks for database services
- Centralized logging and monitoring
- Backup-ready data structure in `/opt/stacks/`
## Security Considerations
### Network Security
- UFW-Docker integration for proper firewall rules
- Services isolated to appropriate network segments
- Restricted access for sensitive tools (Stirling PDF)
### Authentication
- Centralized SSO through Pocket ID for most services
- OAuth integration where supported
- Secure secret management through Ansible vault
### Container Security
- Non-root container execution (UID/GID 1000:1000)
- Security options: `no-new-privileges: true`
- Capability dropping and minimal permissions
## Troubleshooting
### Common Issues
- **Database Connection**: Ensure MySQL containers use proper authentication plugins
- **OAuth Discovery**: Check issuer URLs don't have trailing slashes
- **Migration Failures**: Visit service `/install` endpoints for database setup
- **Network Issues**: Verify containers are on the same Docker network
- **OIDC login fails with a connection error:** the container is missing the
`extra_hosts` entry for `id.thesatelliteoflove.com`.
- **OIDC discovery fails:** check the issuer URL has no trailing slash.
- **Caddy fails with "Address already in use":** another container took
172.20.0.5; the network lacks the reserved range. Run
`migrate-lava-network.yml`.
- **Gitea SSH stops working after a recreate:** Gitea lost its fixed IP;
redeploy with `--tags gitea` to restore it and the firewall rule.