Architecture, README, standards, CLAUDE.md, settings reference, indexes, backlog, release plans and CHANGELOG checked against the code at v2.14.0: backfeed, notifications, counts line and web actions documented; test policy (changed area locally, full suite in CI); stale versions, statuses and links fixed; legacy and superseded docs moved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
23 KiB
StarPunk Architecture Overview
Version: 2.14.0 (2026-11-10) Status: Current. Describes the system as it is in the code today.
Purpose
StarPunk is a minimal, single-user IndieWeb blog/CMS. One person (the site owner) writes short notes, which are published as microformats2-marked HTML pages and as RSS, Atom and JSON feeds. The site takes part in the IndieWeb: it accepts posts from Micropub clients, issues its own IndieAuth tokens for them, sends and receives Webmentions, pushes feed updates with WebSub, and crossposts (POSSE) to GoToSocial, Bluesky and Threads. Responses to those crossposts are pulled back and shown on the note (backfeed), and the owner can be notified of new responses.
Guiding principle: "every line of code must justify its existence". It is a single Flask application with server-side rendering, SQLite and plain files. There is no JavaScript build step, no external database and no message broker. Background work runs in threads inside the web processes.
Related documents
| Topic | Document |
|---|---|
| Dependencies, runtime, tests, CI | technology-stack.md |
| Database schema and migrations | database.md |
| Build, configuration and production deployment | deployment.md |
| Admin login (OIDC) and the built-in IndieAuth server | simplified-auth-architecture.md |
| Micropub, IndieAuth, Webmention, microformats2, WebSub, feeds | indieweb-protocols.md |
| Crossposting (POSSE) to GoToSocial, Bluesky, Threads | syndication-architecture.md |
Key decisions: ADR-064 (OIDC login + built-in IndieAuth server), ADR-063 (Webmention), ADR-065, ADR-066, ADR-069 and ADR-070 (crossposting), ADR-067 (Micropub update/delete), ADR-068 (likes and reposts), ADR-071 (WebSub), ADR-072 (metrics dashboard removed), ADR-073 (backfeed), ADR-074 (notifications), ADR-057 and ADR-058 (media), ADR-034 (search), ADR-052 (configuration), ADR-054 (logging), ADR-055 (errors).
System at a glance
Internet
│
┌─────▼─────┐ TLS, security headers,
│ Caddy │ sets X-Forwarded-For
└─────┬─────┘
│ HTTP :8000
┌─────────────────────────▼─────────────────────────────────────────┐
│ Container: gunicorn, 4 sync worker processes │
│ │
│ each process = Flask app (create_app) + daemon threads: │
│ request handling routes/* blueprints -> Jinja2 templates │
│ webmention worker works its queues only while holding a lock │
│ websub worker works its queues only while holding a lock │
│ syndication worker claims rows with a lease in SQLite │
└───────────────┬──────────────────────────────┬─────────────────────┘
│ │
┌───────▼────────┐ ┌──────────▼──────────────────┐
│ /data volume │ │ Outbound HTTP │
│ starpunk.db │ │ OIDC provider, IndieAuth │
│ notes/*.md │ │ client pages, webmention │
│ media/ │ │ sources/targets, WebSub │
│ avatars/ │ │ callbacks, GoToSocial, │
│ logs/, locks │ │ Bluesky, Threads, avatars, │
└────────────────┘ │ notifications (Apprise) │
└─────────────────────────────┘
Request flow
- Caddy terminates TLS and proxies to gunicorn on port 8000.
app.pybuilds the app once per worker process withstarpunk.create_app()(starpunk/__init__.py). That function loads and validates configuration, configures logging, cleans up old debug and temporary upload files, creates the schema and runs migrations (under a lock, see database.md), creates the connection pool, configures the feed cache and rate limiters, fills the FTS index if it is empty, registers blueprints, error handlers and template helpers, and starts the background threads.before_requestassigns a correlation ID (included in every log line) and, if astarpunk_sessioncookie is present, verifies it and setsg.me.- The blueprint handler runs. Database access goes through
get_db(), which hands out a pooled, request-scoped SQLite connection. after_requesthooks addLink: </webmention>; rel="webmention"to HTML responses, WebSubLink rel=hub/rel=selfheaders to topic responses.- Errors are rendered by the central handlers in
starpunk/errors.py(JSON for Micropub, HTML pages fromtemplates/<code>.htmlelsewhere).
Writing a note (admin form or Micropub) goes through starpunk/notes.py,
which writes the Markdown file and the notes row together. It then updates
the FTS index, queues outgoing webmentions, fetches a reply context for
replies, marks crossposts pending, and tells WebSub what changed (bumping the
site's content_version and queueing deliveries). The HTTP response does not
wait for outbound network calls; the workers make them later.
Components
Application core (starpunk/)
| Module | Responsibility |
|---|---|
__init__.py |
App factory create_app(), logging setup, request hooks, /health, worker start-up, __version__ |
config.py |
Reads environment variables (and .env) into app.config; validates and fails fast |
notes.py |
Note CRUD with file and database kept in sync; soft delete and undelete; triggers follow-up work |
models.py |
Note and other data classes; Markdown rendering and HTML sanitising (nh3 allow-list) |
utils.py |
Slugs, note file paths (notes/YYYY/MM/slug.md), atomic writes, trash, date formatting, absolute_url_for |
slug_utils.py |
Custom slug (mp-slug) validation and timestamp fallback slugs |
markdown_ext.py |
Markdown extension that turns bare URLs into links |
media.py |
Image upload and validation (including HEIC), resizing, variants (thumb/small/medium/large), attachment to notes |
tags.py |
Tag normalisation and note–tag links |
search.py |
FTS5 full-text search (falls back to LIKE if FTS5 is unavailable); index maintained by application code |
auth.py |
Admin login with OpenID Connect (PKCE, nonce, ID-token checks with PyJWT); sessions; require_auth |
indieauth.py |
Built-in IndieAuth authorization server: client discovery, codes, hashed access tokens, scopes |
dev_auth.py |
Development-only login (DEV_MODE=true) |
micropub.py |
Micropub create, update, delete, undelete; queries (q=config, source, syndicate-to, category); photos |
author_discovery.py |
Fetches and caches the owner's h-card from ADMIN_ME (24-hour cache) |
mf2_utils.py |
Helpers for consuming microformats2: representative h-card, responding entry, response type, author |
reply_context.py |
Fetches and caches the post a reply responds to |
webmention_sender.py |
Extracts links from rendered notes and queues outgoing webmentions |
webmention_processor.py |
Validates incoming webmentions: fetch source, parse mf2, classify, sanitise content |
webmention_worker.py |
Background thread working the incoming and outgoing webmention queues; also runs avatar upkeep and the notification sweep |
webmention_queries.py |
Webmention queries for display and moderation |
webmention_ssrf.py |
SSRF-safe fetcher: public-address checks, pinned connections, manual redirects, size and time limits |
websub.py |
WebSub publisher (discovery headers and links, content_version, queueing) and hub request handling |
websub_worker.py |
Background thread: verification of intent, content delivery, publish pings to an external hub |
syndication.py |
POSSE clients (Mastodon API/GoToSocial, Bluesky, Threads), reconcile logic, syndication worker (its loop also calls backfeed.process_due()) |
backfeed.py |
Backfeed: reads responses to crossposts from the silos, stores them as webmentions, schedules checks and backfills |
avatars.py |
Saves response authors' avatars as 96 px WebP files in avatars/; fills missing ones and removes unused ones |
notifications.py |
Notifications through Apprise: the sweep of new responses and immediate problem messages |
csrf.py |
CSRF tokens (HMAC of the session cookie) for admin forms, logout and the IndieAuth consent |
rate_limiter.py |
In-memory sliding-window rate limiter |
errors.py |
Central error handlers |
feeds/ |
rss.py (feedgen), atom.py, json_feed.py, opml.py, negotiation.py (Accept header), cache.py (LRU + TTL feed cache with ETags) |
feed.py |
Backward-compatibility re-export of feeds.rss |
database/ |
schema.py (INITIAL_SCHEMA_SQL), init.py (init_db), pool.py (connection pool) |
migrations.py |
Migration runner for migrations/NNN_*.sql |
Routes (starpunk/routes/)
| Blueprint | File | Paths |
|---|---|---|
public |
public.py |
/, /note/<slug>, /tag/<tag>, /likes, /media/<path>, /avatars/<name>, feeds, /opml.xml |
auth |
auth.py |
/auth/login (GET, POST), /auth/callback, /auth/logout (POST) |
indieauth |
indieauth.py |
/.well-known/oauth-authorization-server, /auth/authorize (GET, POST), /auth/authorize/approve, /auth/token (GET, POST), /auth/revoke |
micropub |
micropub.py |
/micropub (GET, POST), /micropub/media (POST), /media (POST, legacy alias) |
admin |
admin.py |
/admin/… (see below) |
search |
search.py |
/search, /api/search |
webmention |
webmention.py |
/webmention (GET, POST); registered only when WEBMENTION_ENABLED |
websub |
websub.py |
/websub (GET for information, POST to subscribe/unsubscribe) |
dev_auth |
dev_auth.py |
/dev/login; registered only when DEV_MODE=true |
/health is defined directly in create_app().
Templates and static files
Jinja2 templates live in templates/: public pages, admin/, partials/
(for example responses.html for webmentions and reaction.html for likes,
reposts and bookmarks) and one page per HTTP error. templates/base.html
carries the discovery links (rel=micropub, indieauth-metadata,
authorization_endpoint, token_endpoint, webmention, hub/self, me,
feeds). Static assets are one stylesheet (static/css/style.css); admin
pages have small inline scripts. Everything works without JavaScript.
Public URL surface
| Path | Purpose |
|---|---|
/ |
Home page: recent published notes (h-feed); likes are kept off it |
/note/<slug> |
Note permalink (h-entry), with approved responses (webmentions and backfed silo responses), a counts line with web actions, and reply context |
/tag/<tag> |
Notes with a tag |
/likes |
Likes |
/search, /api/search |
Full-text search page and JSON API |
/media/<path> |
Uploaded images and their variants |
/avatars/<name> |
Saved copies of response authors' avatars |
/feed |
Feed chosen by the Accept header (RSS by default; Vary: Accept) |
/feed.rss, /feed.atom, /feed.json |
RSS 2.0, Atom 1.0, JSON Feed 1.1 |
/feed.xml |
Legacy alias of /feed.rss |
/opml.xml |
OPML list of the feeds |
/micropub, /micropub/media |
Micropub and media endpoints (Bearer token) |
/webmention |
Webmention receiver (POST answers 202; validated asynchronously) |
/websub |
WebSub hub for this site's own topics |
/.well-known/oauth-authorization-server, /auth/authorize, /auth/token, /auth/revoke |
IndieAuth server |
/auth/login, /auth/callback, /auth/logout |
Admin login via OIDC |
/admin/ |
Dashboard (notes, WebSub subscriber counts) |
/admin/new, /admin/edit/<id>, /admin/delete/<id> |
Note editor |
/admin/webmentions, /admin/webmentions/<id>, …/action, …/bulk-action |
Webmention moderation |
/admin/tokens, /admin/tokens/<id>/revoke |
Issued IndieAuth tokens |
/admin/syndication/<id>/retry |
Retry a failed crosspost |
/admin/syndication/<id>/check (POST) |
Check one crosspost for responses now |
/admin/backfeed/backfill (POST) |
Check every crosspost for responses once (backfill) |
/admin/notifications/test (POST) |
Send a test notification |
/admin/health |
Full diagnostics (JSON): database, pool statistics, filesystem, disk |
/health |
Public liveness check ({"status": "ok", "version": …}); ?detailed=true requires login |
/dev/login |
Development login; only when DEV_MODE=true |
/, /tag/<tag> and /likes show each note's response counts, linking to
its responses (v2.13.4).
Background workers
Gunicorn runs four worker processes and create_app() runs in each, so each
process starts its own daemon threads. Work that must happen once per site is
coordinated through the data volume:
| Thread | Started when | Coordination | Work |
|---|---|---|---|
Webmention worker (webmention_worker.py) |
WEBMENTION_WORKER_ENABLED and not testing |
Non-blocking fcntl.flock on webmention-worker.lock next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in sending. |
Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries; saves missing response avatars every 10 minutes and removes unused ones weekly (v2.13.0); sweeps new responses into notifications and sets webmentions.notified_at (v2.14.0, notifications.py, ADR-074) |
WebSub worker (websub_worker.py) |
WEBSUB_ENABLED and not testing |
Same pattern with websub-worker.lock |
Verifies subscription intents, delivers topic content (signed with X-Hub-Signature when the subscriber gave a secret), retries, expires leases; pings an external hub instead if WEBSUB_HUB_URL is set |
Syndication worker (syndication.py) |
At least one crosspost target configured, and not testing | No file lock. Rows in syndications are claimed with a 5-minute claimed_until lease by an atomic UPDATE, so any process may work a row but only one at a time; a version column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to threads_token.json. |
Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts. Then, only when BACKFEED_ENABLED, backfeed: checks copies for responses, each claimed with a backfeed_claimed_until lease (v2.13.0, backfeed.py, ADR-073) |
All workers run inside an app context, are daemon threads (they end with the
process) and sleep for their poll interval when there is nothing to do. Under
the test suite (TESTING=True) none of them start; tests call the processing
functions directly.
Data storage
Content is kept in files and everything else in SQLite, both under
DATA_PATH (/data in the container):
- Notes: Markdown files at
notes/YYYY/MM/<slug>.md, written atomically (temporary file, then rename). The file holds the content; thenotesrow holds slug, path, timestamps, published flag, content hash and response properties (in_reply_to,like_of,repost_of,bookmark_of,content_format). Deletion is soft: the row getsdeleted_atand the file moves to.trash/YYYY/MM/, so Micropubundeletecan restore it. - Media:
media/YYYY/MM/<uuid>.<ext>plus<uuid>_<variant>.<ext>variants, staged inmedia/.tmp/while being processed. - Database:
starpunk.db(WAL mode): note metadata, tags, media records, sessions, IndieAuth codes and tokens, webmentions (including backfed silo responses, marked byoriginandremote_id) and the outgoing queue, syndications (with backfeed state and silo-only counts), WebSub subscriptions and queues, caches (author h-card, reply contexts), the FTS5 index andsite_state. See database.md. - Avatars:
avatars/<sha256>.webp, saved copies of response authors' avatars (v2.13.0); they can be re-fetched, so losing them is harmless. - Other files:
logs/starpunk.log(rotating), worker lock files,threads_token.json, anddebug/for failed uploads whenDEBUG_SAVE_FAILED_UPLOADS=true.
The FTS index is maintained by notes.py (SQLite cannot read the note files)
and rebuilt at start-up if it is empty.
Authentication model
There is one user, the site owner, identified by the ADMIN_ME URL. Details
are in simplified-auth-architecture.md.
- Admin login uses OpenID Connect (authorization code with PKCE S256 and a
nonce) against a configured provider (Pocket ID in production). Only the
subject
OIDC_ADMIN_SUBmay log in. The ID token's signature (via JWKS), issuer, audience and expiry are checked. Pending logins are kept inoidc_login_state. - Sessions: a random token in the
starpunk_sessioncookie (HttpOnly, SameSite=Lax, Secure whenSITE_URLis https), stored as a SHA-256 hash insessionsand valid forSESSION_LIFETIMEdays. No passwords are stored. - Micropub tokens: StarPunk is its own IndieAuth authorization server. A
client sends the owner to
/auth/authorize; the owner (logged in through OIDC) approves; the client exchanges the code at/auth/token(PKCE required). Codes and tokens are stored hashed. Tokens lastINDIEAUTH_TOKEN_LIFETIME_DAYSand can be revoked at/auth/revokeor in/admin/tokens. Scopes:create,update,delete,media,profile. - Development:
DEV_MODE=trueadds/dev/login, which logs in asDEV_ADMIN_MEwithout any checks. Never enabled in production; start-up logs a warning when it is on.
Configuration
All configuration comes from environment variables (optionally a .env
file), read once at start-up by starpunk/config.py. Validation fails fast:
SESSION_SECRET is always required; outside DEV_MODE, ADMIN_ME and the
four OIDC_* settings are required too. Crosspost targets switch on when
their credentials are set. The full list with defaults is in
deployment.md.
Caching
- Feed note list: per process, refreshed after
FEED_CACHE_SECONDSor as soon assite_state.content_versionchanges. Every change to a public note bumps that value in the database, so all gunicorn processes notice at once. - Rendered feeds:
feeds/cache.py, an in-memory LRU cache with a TTL (FEED_CACHE_MAX_SIZE,FEED_CACHE_SECONDS), keyed by a checksum of the notes pluscontent_version. Responses carryETagandCache-Control: public, max-age=…; conditional requests get304. - Author h-card (
author_profile, 24 hours) and reply contexts (reply_contexts) are cached in the database. - The webmention worker caches endpoint discovery results in memory.
- Response avatars are saved once as files named by the SHA-256 of the
source URL and served from
/avatars/<name>withCache-Control: public, max-age=604800(one week).
Security posture
- Outbound requests (SSRF): every fetch of a URL that comes from
outside (webmention sources and targets, endpoint discovery, reply
contexts, WebSub callbacks, remote Micropub photos, IndieAuth client pages,
crosspost link cards since v2.11.0, response avatars since v2.13.0) goes
through the fetcher in
webmention_ssrf.py. Every address the host resolves to must be public (private, loopback, link-local, CGNAT, multicast and reserved ranges are refused); it follows redirects by hand and re-checks each hop, connects to the address it checked (no second DNS lookup) and limits response size (1 MB by default) and total time.DEV_MODErelaxes the checks for local testing. Exceptions: avatars on the site's ownGOTOSOCIAL_URLhost are fetched directly, as the crossposting client does, since behind the same proxy it may resolve to an internal address (v2.13.1); Bluesky CDN avatars are fetched from the account's PDS withcom.atproto.sync.getBlob, itself through the safe fetcher. Apprise notifications go to the owner-configuredNOTIFY_URLS, outside the safe fetcher. - HTML sanitising: rendered notes (Markdown or HTML sent by a Micropub
client) and webmention content are cleaned with nh3 allow-lists; search
highlighting escapes everything except
<mark>. - Rate limits:
/webmentionper client IP and per source domain (defaults 60 and 10 per 60 seconds);/websub30 requests per IP per minute, plus limits on subscriptions in total, per callback host and per callback address, and on requests waiting for verification. Request-rate counters live in therate_limitstable, shared by all server processes (v2.11.0). - Client IP behind a proxy: with
TRUSTED_PROXY=truethe first address inX-Forwarded-Foris used as the client IP for rate limiting. Set it only when the app is reachable solely through a proxy that sets that header (Caddy does); without it, every client shares the proxy's address. - Auth: hashed session and token storage, single-use login state, PKCE
for both OIDC and IndieAuth, redirect URIs checked against the client,
post-login redirects limited to this site. Admin form POSTs, logout and
the IndieAuth consent carry a CSRF token: an HMAC of the session cookie,
checked by
starpunk/csrf.py(v2.11.0), on top of the SameSite=Lax cookie. - Uploads: request bodies are capped at 50 MB; images are processed by Pillow and stored under random (UUID) names.
- Container: runs as a non-root user. HTTPS and security headers (HSTS, CSP, X-Frame-Options and so on) come from the reverse proxy.
Observability
- Logs go to stdout and
logs/starpunk.log(10 MB × 10 files); each line carries the request's correlation ID. /healthreports status, version and webmention-worker statistics;/health?detailed=truechecks database, filesystem and disk space;/admin/healthadds pool statistics and filesystem details. (The metrics dashboard was removed in v2.12.0, see ADR-072.)
Last Updated: 2026-11-10 (v2.14.0)