Gondulf/deployment/README.md

# Gondulf Deployment Guide

This guide covers deploying Gondulf IndieAuth Server using OCI-compliant containers with both **Podman** (recommended) and **Docker** (alternative).

## Table of Contents

1. [Quick Start](#quick-start)
2. [Container Engine Support](#container-engine-support)
3. [Prerequisites](#prerequisites)
4. [Building the Container Image](#building-the-container-image)
5. [Development Deployment](#development-deployment)
6. [Production Deployment](#production-deployment)
7. [Backup and Restore](#backup-and-restore)
8. [systemd Integration](#systemd-integration)
9. [Troubleshooting](#troubleshooting)
10. [Security Considerations](#security-considerations)

## Quick Start

### Podman (Rootless - Recommended)

```bash
# 1. Clone and configure
git clone https://github.com/yourusername/gondulf.git
cd gondulf
cp .env.example .env
# Edit .env with your settings

# 2. Build image
podman build -t gondulf:latest .

# 3. Run container
podman run -d --name gondulf \
  -p 8000:8000 \
  -v gondulf_data:/data:Z \
  --env-file .env \
  gondulf:latest

# 4. Verify health
curl http://localhost:8000/health
```

### Docker (Alternative)

```bash
# 1. Clone and configure
git clone https://github.com/yourusername/gondulf.git
cd gondulf
cp .env.example .env
# Edit .env with your settings

# 2. Build and run with compose
docker-compose up -d

# 3. Verify health
curl http://localhost:8000/health
```

## Container Engine Support

Gondulf supports both Podman and Docker with identical functionality.

### Podman (Primary)

**Advantages**:
- Daemonless architecture (no background process)
- Rootless mode for enhanced security
- Native systemd integration
- Pod support for multi-container applications
- OCI-compliant

**Recommended for**: Production deployments, security-focused environments

### Docker (Alternative)

**Advantages**:
- Wide ecosystem and tooling support
- Familiar to most developers
- Extensive documentation

**Recommended for**: Development, existing Docker environments

### Compatibility Matrix

| Feature | Podman | Docker |
|---------|--------|--------|
| Container build | ✅ | ✅ |
| Container runtime | ✅ | ✅ |
| Compose files | ✅ (podman-compose) | ✅ (docker-compose) |
| Rootless mode | ✅ Native | ⚠️ Experimental |
| systemd integration | ✅ Built-in | ⚠️ Manual |
| Health checks | ✅ | ✅ |

## Prerequisites

### System Requirements

- **Operating System**: Linux (recommended), macOS, Windows (WSL2)
- **CPU**: 1 core minimum, 2+ cores recommended
- **RAM**: 512 MB minimum, 1 GB+ recommended
- **Disk**: 5 GB available space

### Container Engine

Choose ONE:

**Option 1: Podman** (Recommended)

```bash
# Fedora/RHEL/CentOS
sudo dnf install podman podman-compose

# Ubuntu/Debian
sudo apt install podman podman-compose

# Verify installation
podman --version
podman-compose --version
```

**Option 2: Docker**

```bash
# Ubuntu/Debian
sudo apt install docker.io docker-compose

# Or install from Docker's repository:
# https://docs.docker.com/engine/install/

# Verify installation
docker --version
docker-compose --version
```

### Rootless Podman Setup (Recommended)

For enhanced security, configure rootless Podman:

```bash
# 1. Check subuid/subgid configuration
grep $USER /etc/subuid
grep $USER /etc/subgid

# Should show: username:100000:65536 (or similar)
# If missing, run:
sudo usermod --add-subuids 100000-165535 $USER
sudo usermod --add-subgids 100000-165535 $USER

# 2. Enable user lingering (services persist after logout)
loginctl enable-linger $USER

# 3. Verify rootless setup
podman system info | grep rootless
# Should show: runRoot: /run/user/1000/...
```

## Building the Container Image

### Using Podman

```bash
# Build image
podman build -t gondulf:latest .

# Verify build
podman images | grep gondulf

# Test run
podman run --rm gondulf:latest python -m gondulf --version
```

### Using Docker

```bash
# Build image
docker build -t gondulf:latest .

# Verify build
docker images | grep gondulf

# Test run
docker run --rm gondulf:latest python -m gondulf --version
```

### Build Arguments

The Dockerfile supports multi-stage builds that include testing:

```bash
# Build with tests (default)
podman build -t gondulf:latest .

# If build fails, tests have failed - check build output
```

## Development Deployment

Development deployment includes:
- Live code reload
- MailHog for local email testing
- Debug logging enabled
- No TLS requirements

### Using Podman Compose

```bash
# Start development environment
podman-compose -f docker-compose.yml -f docker-compose.development.yml up

# Access services:
# - Gondulf: http://localhost:8000
# - MailHog UI: http://localhost:8025

# View logs
podman-compose logs -f gondulf

# Stop environment
podman-compose down
```

### Using Docker Compose

```bash
# Start development environment
docker-compose -f docker-compose.yml -f docker-compose.development.yml up

# Access services:
# - Gondulf: http://localhost:8000
# - MailHog UI: http://localhost:8025

# View logs
docker-compose logs -f gondulf

# Stop environment
docker-compose down
```

### Development Configuration

Create `.env` file from `.env.example`:

```bash
cp .env.example .env
```

Edit `.env` with development settings:

```env
GONDULF_SECRET_KEY=dev-secret-key-minimum-32-characters
GONDULF_BASE_URL=http://localhost:8000
GONDULF_DATABASE_URL=sqlite:///./data/gondulf.db
GONDULF_SMTP_HOST=mailhog
GONDULF_SMTP_PORT=1025
GONDULF_SMTP_USE_TLS=false
GONDULF_DEBUG=true
GONDULF_LOG_LEVEL=DEBUG
```

## Production Deployment

Production deployment includes:
- nginx reverse proxy with TLS termination
- Rate limiting and security headers
- Persistent volume for database
- Health checks and auto-restart
- Proper logging configuration

### Step 1: Configuration

```bash
# 1. Copy environment template
cp .env.example .env

# 2. Generate secret key
python -c "import secrets; print(secrets.token_urlsafe(32))"

# 3. Edit .env with your production settings
nano .env
```

Production `.env` example:

```env
GONDULF_SECRET_KEY=<generated-secret-key-from-step-2>
GONDULF_BASE_URL=https://auth.example.com
GONDULF_DATABASE_URL=sqlite:////data/gondulf.db
GONDULF_SMTP_HOST=smtp.sendgrid.net
GONDULF_SMTP_PORT=587
GONDULF_SMTP_USERNAME=apikey
GONDULF_SMTP_PASSWORD=<your-sendgrid-api-key>
GONDULF_SMTP_FROM=noreply@example.com
GONDULF_SMTP_USE_TLS=true
GONDULF_HTTPS_REDIRECT=true
GONDULF_TRUST_PROXY=true
GONDULF_SECURE_COOKIES=true
GONDULF_DEBUG=false
GONDULF_LOG_LEVEL=INFO
```

### Step 2: TLS Certificates

Obtain TLS certificates (Let's Encrypt recommended):

```bash
# Create SSL directory
mkdir -p deployment/nginx/ssl

# Option 1: Let's Encrypt (recommended)
sudo certbot certonly --standalone -d auth.example.com
sudo cp /etc/letsencrypt/live/auth.example.com/fullchain.pem deployment/nginx/ssl/
sudo cp /etc/letsencrypt/live/auth.example.com/privkey.pem deployment/nginx/ssl/

# Option 2: Self-signed (development/testing only)
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
  -keyout deployment/nginx/ssl/privkey.pem \
  -out deployment/nginx/ssl/fullchain.pem

# Secure permissions
chmod 600 deployment/nginx/ssl/privkey.pem
chmod 644 deployment/nginx/ssl/fullchain.pem
```

### Step 3: nginx Configuration

Edit `deployment/nginx/conf.d/gondulf.conf`:

```nginx
# Change server_name to your domain
server_name auth.example.com;  # ← CHANGE THIS
```

### Step 4: Deploy with Podman (Recommended)

```bash
# Build image
podman build -t gondulf:latest .

# Start services
podman-compose -f docker-compose.yml -f docker-compose.production.yml up -d

# Verify health
curl https://auth.example.com/health

# View logs
podman-compose logs -f
```

### Step 5: Deploy with Docker (Alternative)

```bash
# Build and start
docker-compose -f docker-compose.yml -f docker-compose.production.yml up -d

# Verify health
curl https://auth.example.com/health

# View logs
docker-compose logs -f
```

### Step 6: Verify Deployment

```bash
# 1. Check health endpoint
curl https://auth.example.com/health
# Expected: {"status":"healthy","database":"connected"}

# 2. Check OAuth metadata
curl https://auth.example.com/.well-known/oauth-authorization-server | jq
# Expected: JSON with issuer, authorization_endpoint, token_endpoint

# 3. Verify HTTPS redirect
curl -I http://auth.example.com/
# Expected: 301 redirect to HTTPS

# 4. Check security headers
curl -I https://auth.example.com/ | grep -E "(Strict-Transport|X-Frame|X-Content)"
# Expected: HSTS, X-Frame-Options, X-Content-Type-Options headers

# 5. Test TLS configuration
# Visit: https://www.ssllabs.com/ssltest/analyze.html?d=auth.example.com
# Target: Grade A or higher
```

## Backup and Restore

### Automated Backups

The backup scripts auto-detect Podman or Docker.

#### Create Backup

```bash
# Using included script (works with both Podman and Docker)
./deployment/scripts/backup.sh

# Or with custom backup directory
./deployment/scripts/backup.sh /path/to/backups

# Or using compose (Podman)
podman-compose --profile backup run --rm backup

# Or using compose (Docker)
docker-compose --profile backup run --rm backup
```

Backup details:
- Uses SQLite `VACUUM INTO` for safe hot backups
- No downtime required
- Automatic compression (gzip)
- Integrity verification
- Automatic cleanup of old backups (default: 7 days retention)

#### Scheduled Backups with cron

```bash
# Create cron job for daily backups at 2 AM
crontab -e

# Add this line:
0 2 * * * cd /path/to/gondulf && ./deployment/scripts/backup.sh >> /var/log/gondulf-backup.log 2>&1
```

### Restore from Backup

**CAUTION**: This will replace the current database!

```bash
# Restore from backup
./deployment/scripts/restore.sh /path/to/backups/gondulf_backup_20251120_120000.db.gz

# The script will:
# 1. Stop the container (if running)
# 2. Create a safety backup of current database
# 3. Restore from the specified backup
# 4. Verify integrity
# 5. Restart the container (if it was running)
```

### Test Backup/Restore

```bash
# Run automated backup/restore tests
./deployment/scripts/test-backup-restore.sh

# This verifies:
# - Backup creation
# - Backup integrity
# - Database structure
# - Compression
# - Queryability
```

## systemd Integration

### Rootless Podman (Recommended)

**Method 1: Podman-Generated Unit** (Recommended)

```bash
# 1. Start container normally first
podman run -d --name gondulf \
  -p 8000:8000 \
  -v gondulf_data:/data:Z \
  --env-file /home/$USER/gondulf/.env \
  gondulf:latest

# 2. Generate systemd unit file
cd ~/.config/systemd/user/
podman generate systemd --new --files --name gondulf

# 3. Stop the manually-started container
podman stop gondulf
podman rm gondulf

# 4. Enable and start service
systemctl --user daemon-reload
systemctl --user enable --now container-gondulf.service

# 5. Enable lingering (service runs without login)
loginctl enable-linger $USER

# 6. Verify status
systemctl --user status container-gondulf
```

**Method 2: Custom Unit File**

```bash
# 1. Copy unit file
mkdir -p ~/.config/systemd/user/
cp deployment/systemd/gondulf-podman.service ~/.config/systemd/user/gondulf.service

# 2. Edit paths if needed
nano ~/.config/systemd/user/gondulf.service

# 3. Reload and enable
systemctl --user daemon-reload
systemctl --user enable --now gondulf.service
loginctl enable-linger $USER

# 4. Verify status
systemctl --user status gondulf
```

**systemd User Service Commands**:

```bash
# Start service
systemctl --user start gondulf

# Stop service
systemctl --user stop gondulf

# Restart service
systemctl --user restart gondulf

# Check status
systemctl --user status gondulf

# View logs
journalctl --user -u gondulf -f

# Disable service
systemctl --user disable gondulf
```

### Docker (System Service)

```bash
# 1. Copy unit file
sudo cp deployment/systemd/gondulf-docker.service /etc/systemd/system/gondulf.service

# 2. Edit paths in the file
sudo nano /etc/systemd/system/gondulf.service
# Change WorkingDirectory to your installation path

# 3. Reload and enable
sudo systemctl daemon-reload
sudo systemctl enable --now gondulf.service

# 4. Verify status
sudo systemctl status gondulf
```

**systemd System Service Commands**:

```bash
# Start service
sudo systemctl start gondulf

# Stop service
sudo systemctl stop gondulf

# Restart service
sudo systemctl restart gondulf

# Check status
sudo systemctl status gondulf

# View logs
sudo journalctl -u gondulf -f

# Disable service
sudo systemctl disable gondulf
```

### Compose-Based systemd Service

For deploying with docker-compose or podman-compose:

```bash
# For Podman (rootless):
cp deployment/systemd/gondulf-compose.service ~/.config/systemd/user/gondulf.service
# Edit to use podman-compose
systemctl --user daemon-reload
systemctl --user enable --now gondulf.service

# For Docker (rootful):
sudo cp deployment/systemd/gondulf-compose.service /etc/systemd/system/gondulf.service
# Edit to use docker-compose and add docker.service dependency
sudo systemctl daemon-reload
sudo systemctl enable --now gondulf.service
```

## Troubleshooting

### Container Won't Start

**Check logs**:

```bash
# Podman
podman logs gondulf
# or
podman-compose logs gondulf

# Docker
docker logs gondulf
# or
docker-compose logs gondulf
```

**Common issues**:

1. **Missing SECRET_KEY**:
   ```
   ERROR: GONDULF_SECRET_KEY is required
   ```
   Solution: Set `GONDULF_SECRET_KEY` in `.env` (minimum 32 characters)

2. **Missing BASE_URL**:
   ```
   ERROR: GONDULF_BASE_URL is required
   ```
   Solution: Set `GONDULF_BASE_URL` in `.env`

3. **Port already in use**:
   ```
   Error: bind: address already in use
   ```
   Solution:
   ```bash
   # Check what's using port 8000
   sudo ss -tlnp | grep 8000

   # Use different port
   podman run -p 8001:8000 ...
   ```

### Database Issues

**Check database file**:

```bash
# Podman
podman exec gondulf ls -la /data/

# Docker
docker exec gondulf ls -la /data/
```

**Check database integrity**:

```bash
# Podman
podman exec gondulf sqlite3 /data/gondulf.db "PRAGMA integrity_check;"

# Docker
docker exec gondulf sqlite3 /data/gondulf.db "PRAGMA integrity_check;"
```

**Expected output**: `ok`

### Permission Errors (Rootless Podman)

If you see permission errors with volumes:

```bash
# 1. Check subuid/subgid configuration
grep $USER /etc/subuid
grep $USER /etc/subgid

# 2. Add if missing
sudo usermod --add-subuids 100000-165535 $USER
sudo usermod --add-subgids 100000-165535 $USER

# 3. Restart user services
systemctl --user daemon-reload

# 4. Use :Z label for SELinux systems
podman run -v ./data:/data:Z ...
```

### SELinux Issues

On SELinux-enabled systems (RHEL, Fedora, CentOS):

```bash
# Check for SELinux denials
sudo ausearch -m AVC -ts recent

# Solution 1: Add :Z label to volumes (recommended)
podman run -v gondulf_data:/data:Z ...

# Solution 2: Temporarily permissive (testing only)
sudo setenforce 0

# Solution 3: Create SELinux policy (advanced)
# Use audit2allow to generate policy from denials
```

### Email Not Sending

**Check SMTP configuration**:

```bash
# Test SMTP connection from container
podman exec gondulf sh -c "timeout 5 bash -c '</dev/tcp/smtp.example.com/587' && echo 'Port open' || echo 'Port closed'"

# Check logs for SMTP errors
podman logs gondulf | grep -i smtp
```

**Common SMTP issues**:

1. **Authentication failure**: Verify username/password (use app-specific password for Gmail)
2. **TLS error**: Check `GONDULF_SMTP_USE_TLS` matches port (587=STARTTLS, 465=TLS, 25=none)
3. **Firewall**: Ensure outbound connections allowed on SMTP port

### Health Check Failing

```bash
# Check health status
podman inspect gondulf --format='{{.State.Health.Status}}'

# View health check logs
podman inspect gondulf --format='{{range .State.Health.Log}}{{.Output}}{{end}}'

# Test health endpoint manually
curl http://localhost:8000/health
```

### nginx Issues

**Test nginx configuration**:

```bash
# Podman
podman exec gondulf_nginx nginx -t

# Docker
docker exec gondulf_nginx nginx -t
```

**Check nginx logs**:

```bash
# Podman
podman logs gondulf_nginx

# Docker
docker logs gondulf_nginx
```

## Security Considerations

### Container Security (Rootless Podman)

Rootless Podman provides defense-in-depth:

- No root daemon
- User namespace isolation
- UID mapping (container UID 1000 → host subuid range)
- Limited attack surface

### TLS/HTTPS Requirements

IndieAuth **requires HTTPS in production**:

- Obtain valid TLS certificate (Let's Encrypt recommended)
- Configure nginx for TLS termination
- Enable HSTS headers
- Use strong ciphers (TLS 1.2+)

### Secrets Management

**Never commit secrets to version control**:

```bash
# Verify .env is gitignored
git check-ignore .env
# Should output: .env

# Ensure .env has restrictive permissions
chmod 600 .env
```

**Production secrets best practices**:
- Use strong SECRET_KEY (32+ characters)
- Use app-specific passwords for email (Gmail, etc.)
- Rotate secrets regularly
- Consider secrets management tools (Vault, AWS Secrets Manager)

### Network Security

**Firewall configuration**:

```bash
# Allow HTTPS (443)
sudo ufw allow 443/tcp

# Allow HTTP (80) for Let's Encrypt challenges and redirects
sudo ufw allow 80/tcp

# Block direct access to container port (8000)
# Don't expose port 8000 externally in production
```

### Rate Limiting

nginx configuration includes rate limiting:
- Authorization endpoint: 10 req/s (burst 20)
- Token endpoint: 20 req/s (burst 40)
- General endpoints: 30 req/s (burst 60)

Adjust in `deployment/nginx/conf.d/gondulf.conf` as needed.

### Security Headers

The following security headers are automatically set:
- `Strict-Transport-Security` (HSTS)
- `X-Frame-Options: DENY`
- `X-Content-Type-Options: nosniff`
- `X-XSS-Protection: 1; mode=block`
- `Referrer-Policy`
- Content-Security-Policy (set by application)

### Regular Security Updates

```bash
# Update base image
podman pull python:3.12-slim-bookworm

# Rebuild container
podman build -t gondulf:latest .

# Recreate container
podman stop gondulf
podman rm gondulf
podman run -d --name gondulf ...
```

## Additional Resources

- [Gondulf Documentation](../docs/)
- [Podman Documentation](https://docs.podman.io/)
- [Docker Documentation](https://docs.docker.com/)
- [W3C IndieAuth Specification](https://www.w3.org/TR/indieauth/)
- [Let's Encrypt](https://letsencrypt.org/)
- [Rootless Containers](https://rootlesscontaine.rs/)

## Support

For issues or questions:
- GitHub Issues: https://github.com/yourusername/gondulf/issues
- Documentation: https://github.com/yourusername/gondulf/docs
- Security: security@yourdomain.com