ansible/todo.md

# Infrastructure Improvements TODO

## High Priority (Quick Wins)

### 1. Split the massive docker role ✅ COMPLETED
- **Current Issue**: `roles/docker/tasks/main.yml` has 20+ services in one file (176 lines)
- **Solution**: Break into logical service groups:
  ```
  roles/docker/tasks/
  ├── main.yml (orchestrator)
  ├── infrastructure/ (caddy, pocketid, dockge)
  ├── development/ (gitea, codeserver, bytestash)
  ├── media/ (audiobookshelf, kavita, ghost, pinchflat, pinry, hoarder, manyfold)
  ├── productivity/ (paperless, baikal, syncthing, mmdl, heyform, dawarich, palmr, obsidian-livesync)
  ├── communication/ (gotosocial, postiz)
  └── monitoring/ (changedetection, appriseapi, gotify)
  ```
- **COMPLETED**: All services organized into logical categories with category-level tags

### 2. Standardize variable management ✅ COMPLETED
- **Current Issue**: Secrets in single encrypted file, no clear variable hierarchy
- **Solution**: Create proper variable structure:
  ```
  group_vars/
  ├── all/
  │   ├── domains.yml (domain and subdomain mappings)
  │   ├── infrastructure.yml (network config, Docker settings)
  │   ├── vault.yml (encrypted secrets with vault_ prefix)
  │   └── services.yml (service configuration flags)
  ```
- **COMPLETED**: Implemented comprehensive variable hierarchy, updated all templates to use centralized variables, fixed service tag isolation

### 3. Template consolidation ✅ PARTIALLY COMPLETED
- **Current Issue**: Many compose templates repeat patterns, some services used static files
- **Solution**: Create reusable template includes with standard service template structure
- **COMPLETED**: Converted all static compose files (caddy, dockge, hoarder) to Jinja2 templates
- **REMAINING**: Create reusable template patterns for common configurations

## Security & Reliability

### 4. Add health checks
- **Issue**: Most services lack proper healthcheck configurations in compose templates
- **Solution**: Implement comprehensive health monitoring with standardized healthcheck patterns

### 5. Implement backup strategy
- **Issue**: No automated backups for 25 services and their data
- **Solution**: Add backup role with:
  - Database dumps for PostgreSQL services
  - Volume backups for file-based services
  - Rotation policies
  - Restoration testing

### 6. Network segmentation
- **Issue**: All services share one Docker network
- **Solution**: Separate into:
  - `frontend` (Public-facing services)
  - `backend` (Internal services only)
  - `database` (Database access only)

### 7. Security hardening
- Remove unnecessary `user: root` from services
- Add security contexts to all containers
- Implement least-privilege access patterns
- Add fail2ban for authentication services

## Automation Opportunities

### 8. CI/CD with Gitea Actions
- Leverage self-hosted Gitea for:
  - Ansible syntax validation
  - Service configuration testing
  - Automated deployment triggers
  - Rollback capabilities

### 9. Configuration drift detection
- Add validation tasks to catch manual changes
- Implement configuration validation with proper assertions

### 10. Service dependency management
- **Issue**: Some services depend on Pocket ID SSO but no startup ordering
- **Solution**: Implement dependency checking and startup ordering

### 11. Ansible best practices
- Replace deprecated `apt_key` with proper patterns
- Use `ansible.builtin` FQCN consistently
- Add `check_mode` support
- Implement proper idempotency checks

### 12. Documentation automation
- Auto-generate service inventory
- Create service documentation templates
- Implement automated documentation updates

## Implementation Roadmap

### Week 1: Foundation
- [x] Document improvements in todo.md
- [x] Reorganize docker role structure
- [x] Convert static compose files to templates
- [x] Remove unused services (beaver, grist, stirlingpdf, tasksmd, redlib)
- [x] Clean up templates and files directories
- [x] Implement variable hierarchy
- [ ] Create reusable template patterns

### Week 2: Security & Monitoring  
- [ ] Add health checks
- [ ] Implement backup strategy
- [ ] Security hardening

### Week 3: Automation
- [ ] CI/CD pipeline setup
- [ ] Configuration validation
- [ ] Documentation automation

### Week 4: Advanced Features
- [ ] Network segmentation
- [ ] Dependency management
- [ ] Monitoring dashboard

## Completed Work Summary

### ✅ Major Accomplishments
- **Docker Role Reorganization**: Split monolithic 176-line main.yml into 6 logical service categories
- **Template Standardization**: Converted all static compose files to Jinja2 templates
- **Service Cleanup**: Removed 5 unused/broken services (beaver, grist, stirlingpdf, tasksmd, redlib)
- **Category-Based Deployment**: Can now deploy services by category using tags (infrastructure, media, etc.)
- **Variable Management**: Implemented comprehensive centralized variable hierarchy with proper secret organization
- **Service Tag Isolation**: Fixed service tags to deploy individual services only (not entire categories)
- **Documentation Updates**: Updated all README files and CLAUDE.md to reflect new architecture

### 📊 Current Stats
- **25 active services** organized into 6 categories
- **100% templated** compose files (no static files)
- **6 service directories** for logical organization
- **Clean file structure** with only essential static files

## Notes
- Current architecture is solid and much better organized for long-term maintainability
- Focus on high-impact, low-effort improvements first
- Leverage existing infrastructure (Gitea, Pocket ID) for automation
- Template-driven approach enables future dynamic configuration