Add guard rails for several agents working on the repo at once
Deploys from unpushed branches were being reverted by the next deploy from main, and in-place vault edits could overwrite each other. - deploy-guard.yml (imported by site.yml and dns.yml) refuses real runs unless the checkout is clean and exactly origin/main; --check is fine. - bin/deploy holds a lock so only one deploy runs at a time. - bin/vault-run edits the vault under a lock with an atomic swap. - bin/worktree starts a task in a fresh worktree from origin/main. - CLAUDE.md explains the workflow. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Phil
@@ -8,6 +8,37 @@ This is a personal infrastructure Ansible playbook that automates deployment and
|
||||
|
||||
**Important**: Always review `DEPLOYMENT_LEARNINGS.md` when working on this repository for lessons learned and troubleshooting guidance.
|
||||
|
||||
## Working alongside other agents (read first)
|
||||
|
||||
Several agents change this repo and deploy to the same server at once. Two
|
||||
things went wrong before these rules: a deploy from an unpushed branch was
|
||||
silently reverted by the next deploy from main, and vault edits overwrote each
|
||||
other. Follow these steps:
|
||||
|
||||
1. **Never edit the main checkout.** Start each task in its own worktree:
|
||||
`bin/worktree <branch-name>` (based on the latest origin/main, with the
|
||||
vault files linked in).
|
||||
2. **Land on main before deploying.** Commit, `git fetch && git rebase
|
||||
origin/main`, `git push origin HEAD:main`, then deploy that exact commit.
|
||||
`deploy-guard.yml` (imported by site.yml and dns.yml) refuses real runs from
|
||||
a checkout that is dirty or isn't origin/main. `--check` runs are always
|
||||
allowed; `-e allow_unpushed=true` is for emergencies only, and the fix must
|
||||
be pushed straight after.
|
||||
3. **Deploy with `bin/deploy`**, not ansible-playbook directly. It holds a lock
|
||||
so only one deploy runs at a time (`bin/deploy --tags <service>`;
|
||||
`PLAYBOOK=dns.yml bin/deploy` for DNS). Use the narrowest tag that covers
|
||||
your change.
|
||||
4. **Edit the vault only with `bin/vault-run <command>`.** It locks, backs up,
|
||||
decrypts to a private temp file passed as `$VAULT_PLAIN`, checks the YAML,
|
||||
re-encrypts and swaps it in at once. Never `ansible-vault edit`, and don't
|
||||
decrypt vault.yml in place.
|
||||
5. **Check the live result after deploying** (e.g. `docker inspect`, a login
|
||||
test). If someone else's deploy undid yours, rebase, push, and redeploy
|
||||
rather than deploying an older checkout.
|
||||
6. **Shared files** (Caddyfile.j2, domains.yml, dns.yml, main.yml task lists)
|
||||
are touched by everyone: keep edits small, rebase just before pushing, and
|
||||
resolve conflicts by keeping both sides.
|
||||
|
||||
## Common Commands
|
||||
|
||||
### Initial Setup
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
# Shared by the bin/ helpers. The main checkout holds the venv and the
|
||||
# (gitignored) vault files that every worktree uses.
|
||||
repo=$(git rev-parse --show-toplevel)
|
||||
main_checkout=$(git -C "$repo" worktree list --porcelain | awk '/^worktree /{print $2; exit}')
|
||||
lock_dir=${XDG_RUNTIME_DIR:-/tmp}/ansible-selfhosting
|
||||
mkdir -p "$lock_dir" && chmod 700 "$lock_dir"
|
||||
|
||||
# take_lock NAME: wait for, then hold, a lock shared by every agent on this machine.
|
||||
take_lock() {
|
||||
local name=$1
|
||||
exec 9>"$lock_dir/$name.lock"
|
||||
if ! flock -n 9; then
|
||||
echo "Waiting for $name lock, held by: $(cat "$lock_dir/$name.owner" 2>/dev/null || echo unknown)" >&2
|
||||
flock 9
|
||||
fi
|
||||
echo "$(basename "$repo") (pid $$, $(date '+%H:%M:%S'))" > "$lock_dir/$name.owner"
|
||||
}
|
||||
|
||||
# Worktrees need the gitignored vault files from the main checkout.
|
||||
link_vault() {
|
||||
[ "$repo" = "$main_checkout" ] && return
|
||||
[ -e "$repo/group_vars/all/vault.yml" ] || ln -s "$main_checkout/group_vars/all/vault.yml" "$repo/group_vars/all/vault.yml"
|
||||
[ -e "$repo/vault_pass" ] || ln -s "$main_checkout/vault_pass" "$repo/vault_pass"
|
||||
}
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run a playbook while holding the deploy lock, so only one agent deploys at a
|
||||
# time. Arguments go straight to ansible-playbook.
|
||||
# bin/deploy --tags gitea (site.yml)
|
||||
# bin/deploy --check --diff
|
||||
# PLAYBOOK=dns.yml bin/deploy
|
||||
# Real runs only work from a clean checkout of origin/main (deploy-guard.yml).
|
||||
set -euo pipefail
|
||||
. "$(dirname "$0")/_common.sh"
|
||||
link_vault
|
||||
take_lock deploy
|
||||
cd "$repo"
|
||||
"$main_checkout/.venv/bin/ansible-playbook" "${PLAYBOOK:-site.yml}" -i hosts.yml \
|
||||
--vault-password-file "$main_checkout/vault_pass" "$@"
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
# Edit the shared vault safely: lock, back up, decrypt to a private temp file,
|
||||
# run your command on it, check the YAML, re-encrypt, and swap it in at once.
|
||||
# The command gets the plaintext path as $VAULT_PLAIN (also appended as the
|
||||
# last argument). Never use `ansible-vault edit`.
|
||||
# bin/vault-run python3 my_edit.py
|
||||
# bin/vault-run nvim
|
||||
set -euo pipefail
|
||||
. "$(dirname "$0")/_common.sh"
|
||||
[ $# -gt 0 ] || { echo "usage: bin/vault-run <command...>" >&2; exit 2; }
|
||||
vault="$main_checkout/group_vars/all/vault.yml"
|
||||
pass="$main_checkout/vault_pass"
|
||||
av="$main_checkout/.venv/bin/ansible-vault"
|
||||
take_lock vault
|
||||
backups="$HOME/Documents/backups/ansible-vault"; mkdir -p "$backups"
|
||||
cp -p "$vault" "$backups/vault.yml.$(date +%Y%m%d-%H%M%S)"
|
||||
tmp=$(mktemp -d "$lock_dir/vault.XXXXXX"); trap 'shred -u "$tmp"/* 2>/dev/null; rm -rf "$tmp"' EXIT
|
||||
export VAULT_PLAIN="$tmp/vault.yml"
|
||||
"$av" decrypt --vault-password-file "$pass" --output "$VAULT_PLAIN" "$vault" >/dev/null
|
||||
"$@" "$VAULT_PLAIN"
|
||||
"$main_checkout/.venv/bin/python" -c 'import sys,yaml; yaml.safe_load(open(sys.argv[1]))' "$VAULT_PLAIN"
|
||||
"$av" encrypt --vault-password-file "$pass" --output "$tmp/vault.enc" "$VAULT_PLAIN" >/dev/null
|
||||
chmod --reference="$vault" "$tmp/vault.enc" 2>/dev/null || true
|
||||
mv "$tmp/vault.enc" "$vault"
|
||||
echo "vault updated"
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# Start work on a fresh branch in its own worktree, based on the latest main.
|
||||
# bin/worktree <branch-name>
|
||||
set -euo pipefail
|
||||
. "$(dirname "$0")/_common.sh"
|
||||
name=${1:?usage: bin/worktree <branch-name>}
|
||||
dest="$(dirname "$main_checkout")/ansible-worktrees/$name"
|
||||
git -C "$main_checkout" fetch -q origin
|
||||
git -C "$main_checkout" worktree add -b "$name" "$dest" origin/main
|
||||
cd "$dest" && repo=$dest && link_vault
|
||||
echo "Ready: $dest"
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
# Imported at the top of site.yml and dns.yml.
|
||||
#
|
||||
# Several agents work on this repo at once. If one deploys from an unpushed
|
||||
# branch, the next deploy from main quietly puts the old config back. So a
|
||||
# real run only goes ahead when this checkout is exactly origin/main:
|
||||
# land your commit on main first, then deploy that commit.
|
||||
#
|
||||
# --check runs are always allowed. -e allow_unpushed=true skips the guard
|
||||
# for emergencies; push the fix to main straight after.
|
||||
- name: Only deploy what's on origin/main
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
tags: always
|
||||
tasks:
|
||||
- name: Check this checkout matches origin/main
|
||||
ansible.builtin.shell: |
|
||||
set -eu
|
||||
git fetch -q origin main
|
||||
if [ -n "$(git status --porcelain --untracked-files=no)" ]; then
|
||||
echo "This checkout has uncommitted changes. Commit them, land them on main, then deploy."
|
||||
exit 1
|
||||
fi
|
||||
head=$(git rev-parse HEAD); main=$(git rev-parse origin/main)
|
||||
if [ "$head" != "$main" ]; then
|
||||
echo "HEAD $(git rev-parse --short HEAD) isn't origin/main ($(git rev-parse --short origin/main))."
|
||||
if git merge-base --is-ancestor "$main" "$head"; then
|
||||
echo "Your commits aren't on main yet: rebase, push to main, then deploy."
|
||||
else
|
||||
echo "main has moved on: rebase onto origin/main first, or other agents' changes get reverted."
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
args:
|
||||
chdir: "{{ playbook_dir }}"
|
||||
changed_when: false
|
||||
check_mode: false
|
||||
when:
|
||||
- not ansible_check_mode
|
||||
- not (allow_unpushed | default(false) | bool)
|
||||
@@ -1,5 +1,7 @@
|
||||
---
|
||||
# dns.yml
|
||||
- import_playbook: deploy-guard.yml
|
||||
|
||||
- name: Add A Records for thesatelliteoflove.com and nerder.land
|
||||
hosts: localhost
|
||||
gather_facts: false
|
||||
|
||||
Reference in New Issue
Block a user