- architecture/: overview, technology stack, auth (OIDC + built-in IndieAuth), syndication (POSSE) rewritten; new database.md, deployment.md and indieweb-protocols.md; legacy stack marked historical - decisions/: reconstructed ADR-060 and ADR-063; 31 status corrections (superseded auth ADRs, proposed -> accepted); INDEX rebuilt - design/: reconstructed the v1.6.0 documents the webmention code cites; reports for 18 v2 releases without a design folder; INDEX rebuilt - projectplan/: BACKLOG rewritten (shipped items removed, open issues from the reviews added); RELEASE.md for v2.0.0-v2.10.1; INDEX - standards/: development setup (uv, tests, DEV_MODE), testing (CI test stage), git and release flow, versioning, coding standards, cookies - README, CLAUDE.md, .env.example (every setting the code reads, with defaults) brought up to date - 137 broken links repaired (old absolute paths, moved files); stale ADR references in code comments corrected (comments only) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
22 KiB
StarPunk Architecture Overview
Version: 2.10.1 (2026-11-03) Status: Current. Describes the system as it is in the code today.
Purpose
StarPunk is a minimal, single-user IndieWeb blog/CMS. One person (the site owner) writes short notes, which are published as microformats2-marked HTML pages and as RSS, Atom and JSON feeds. The site takes part in the IndieWeb: it accepts posts from Micropub clients, issues its own IndieAuth tokens for them, sends and receives Webmentions, pushes feed updates with WebSub, and crossposts (POSSE) to GoToSocial, Bluesky and Threads.
Guiding principle: "every line of code must justify its existence". It is a single Flask application with server-side rendering, SQLite and plain files. There is no JavaScript build step, no external database and no message broker. Background work runs in threads inside the web processes.
Related documents
| Topic | Document |
|---|---|
| Dependencies, runtime, tests, CI | technology-stack.md |
| Database schema and migrations | database.md |
| Build, configuration and production deployment | deployment.md |
| Admin login (OIDC) and the built-in IndieAuth server | simplified-auth-architecture.md |
| Micropub, IndieAuth, Webmention, microformats2, WebSub, feeds | indieweb-protocols.md |
| Crossposting (POSSE) to GoToSocial, Bluesky, Threads | syndication-architecture.md |
Key decisions: ADR-064 (OIDC login + built-in IndieAuth server), ADR-063 (Webmention), ADR-065, ADR-066, ADR-069 and ADR-070 (crossposting), ADR-067 (Micropub update/delete), ADR-068 (likes and reposts), ADR-071 (WebSub), ADR-057 and ADR-058 (media), ADR-034 (search), ADR-052 (configuration), ADR-053 and ADR-060 (metrics), ADR-054 (logging), ADR-055 (errors).
System at a glance
Internet
│
┌─────▼─────┐ TLS, security headers,
│ Caddy │ sets X-Forwarded-For
└─────┬─────┘
│ HTTP :8000
┌─────────────────────────▼─────────────────────────────────────────┐
│ Container: gunicorn, 4 sync worker processes │
│ │
│ each process = Flask app (create_app) + daemon threads: │
│ request handling routes/* blueprints -> Jinja2 templates │
│ webmention worker works its queues only while holding a lock │
│ websub worker works its queues only while holding a lock │
│ syndication worker claims rows with a lease in SQLite │
│ memory monitor per-process metrics │
└───────────────┬──────────────────────────────┬─────────────────────┘
│ │
┌───────▼────────┐ ┌──────────▼──────────────────┐
│ /data volume │ │ Outbound HTTP │
│ starpunk.db │ │ OIDC provider, IndieAuth │
│ notes/*.md │ │ client pages, webmention │
│ media/ │ │ sources/targets, WebSub │
│ logs/, locks │ │ callbacks, GoToSocial, │
└────────────────┘ │ Bluesky, Threads │
└─────────────────────────────┘
Request flow
- Caddy terminates TLS and proxies to gunicorn on port 8000.
app.pybuilds the app once per worker process withstarpunk.create_app()(starpunk/__init__.py). That function loads and validates configuration, configures logging, cleans up old debug and temporary upload files, creates the schema and runs migrations (under a lock, see database.md), creates the connection pool, installs the HTTP metrics middleware, configures the feed cache and rate limiters, fills the FTS index if it is empty, registers blueprints, error handlers and template helpers, and starts the background threads.before_requestassigns a correlation ID (included in every log line) and, if astarpunk_sessioncookie is present, verifies it and setsg.me.- The blueprint handler runs. Database access goes through
get_db(), which hands out a pooled, request-scoped SQLite connection. after_requesthooks addLink: </webmention>; rel="webmention"to HTML responses, WebSubLink rel=hub/rel=selfheaders to topic responses, and request metrics with anX-Request-IDheader.- Errors are rendered by the central handlers in
starpunk/errors.py(JSON for Micropub, HTML pages fromtemplates/<code>.htmlelsewhere).
Writing a note (admin form or Micropub) goes through starpunk/notes.py,
which writes the Markdown file and the notes row together. It then updates
the FTS index, queues outgoing webmentions, fetches a reply context for
replies, marks crossposts pending, and tells WebSub what changed (bumping the
site's content_version and queueing deliveries). The HTTP response does not
wait for outbound network calls; the workers make them later.
Components
Application core (starpunk/)
| Module | Responsibility |
|---|---|
__init__.py |
App factory create_app(), logging setup, request hooks, /health, worker start-up, __version__ |
config.py |
Reads environment variables (and .env) into app.config; validates and fails fast |
notes.py |
Note CRUD with file and database kept in sync; soft delete and undelete; triggers follow-up work |
models.py |
Note and other data classes; Markdown rendering and HTML sanitising (nh3 allow-list) |
utils.py |
Slugs, note file paths (notes/YYYY/MM/slug.md), atomic writes, trash, date formatting, absolute_url_for |
slug_utils.py |
Custom slug (mp-slug) validation and timestamp fallback slugs |
markdown_ext.py |
Markdown extension that turns bare URLs into links |
media.py |
Image upload and validation (including HEIC), resizing, variants (thumb/small/medium/large), attachment to notes |
tags.py |
Tag normalisation and note–tag links |
search.py |
FTS5 full-text search (falls back to LIKE if FTS5 is unavailable); index maintained by application code |
auth.py |
Admin login with OpenID Connect (PKCE, nonce, ID-token checks with PyJWT); sessions; require_auth |
indieauth.py |
Built-in IndieAuth authorization server: client discovery, codes, hashed access tokens, scopes |
dev_auth.py |
Development-only login (DEV_MODE=true) |
micropub.py |
Micropub create, update, delete, undelete; queries (q=config, source, syndicate-to, category); photos |
author_discovery.py |
Fetches and caches the owner's h-card from ADMIN_ME (24-hour cache) |
mf2_utils.py |
Helpers for consuming microformats2: representative h-card, responding entry, response type, author |
reply_context.py |
Fetches and caches the post a reply responds to |
webmention_sender.py |
Extracts links from rendered notes and queues outgoing webmentions |
webmention_processor.py |
Validates incoming webmentions: fetch source, parse mf2, classify, sanitise content |
webmention_worker.py |
Background thread working the incoming and outgoing webmention queues |
webmention_queries.py |
Webmention queries for display and moderation |
webmention_ssrf.py |
SSRF-safe fetcher: public-address checks, pinned connections, manual redirects, size and time limits |
websub.py |
WebSub publisher (discovery headers and links, content_version, queueing) and hub request handling |
websub_worker.py |
Background thread: verification of intent, content delivery, publish pings to an external hub |
syndication.py |
POSSE clients (Mastodon API/GoToSocial, Bluesky, Threads), reconcile logic, syndication worker |
rate_limiter.py |
In-memory sliding-window rate limiter |
errors.py |
Central error handlers |
feeds/ |
rss.py (feedgen), atom.py, json_feed.py, opml.py, negotiation.py (Accept header), cache.py (LRU + TTL feed cache with ETags) |
feed.py |
Backward-compatibility re-export of feeds.rss |
monitoring/ |
Metrics buffer, HTTP middleware, database timing wrapper, memory monitor, business metrics |
database/ |
schema.py (INITIAL_SCHEMA_SQL), init.py (init_db), pool.py (connection pool) |
migrations.py |
Migration runner for migrations/NNN_*.sql |
Routes (starpunk/routes/)
| Blueprint | File | Paths |
|---|---|---|
public |
public.py |
/, /note/<slug>, /tag/<tag>, /likes, /media/<path>, feeds, /opml.xml |
auth |
auth.py |
/auth/login (GET, POST), /auth/callback, /auth/logout (POST) |
indieauth |
indieauth.py |
/.well-known/oauth-authorization-server, /auth/authorize (GET, POST), /auth/authorize/approve, /auth/token (GET, POST), /auth/revoke |
micropub |
micropub.py |
/micropub (GET, POST), /micropub/media (POST), /media (POST, legacy alias) |
admin |
admin.py |
/admin/… (see below) |
search |
search.py |
/search, /api/search |
webmention |
webmention.py |
/webmention (GET, POST); registered only when WEBMENTION_ENABLED |
websub |
websub.py |
/websub (GET for information, POST to subscribe/unsubscribe) |
dev_auth |
dev_auth.py |
/dev/login; registered only when DEV_MODE=true |
/health is defined directly in create_app().
Templates and static files
Jinja2 templates live in templates/: public pages, admin/, partials/
(for example responses.html for webmentions and reaction.html for likes,
reposts and bookmarks) and one page per HTTP error. templates/base.html
carries the discovery links (rel=micropub, indieauth-metadata,
authorization_endpoint, token_endpoint, webmention, hub/self, me,
feeds). Static assets are one stylesheet (static/css/style.css) and one
script (static/js/preview.js, the Markdown preview in the admin editor).
Everything works without JavaScript.
Public URL surface
| Path | Purpose |
|---|---|
/ |
Home page: recent published notes (h-feed); likes are kept off it |
/note/<slug> |
Note permalink (h-entry), with approved webmentions and reply context |
/tag/<tag> |
Notes with a tag |
/likes |
Likes |
/search, /api/search |
Full-text search page and JSON API |
/media/<path> |
Uploaded images and their variants |
/feed |
Feed chosen by the Accept header (RSS by default; Vary: Accept) |
/feed.rss, /feed.atom, /feed.json |
RSS 2.0, Atom 1.0, JSON Feed 1.1 |
/feed.xml |
Legacy alias of /feed.rss |
/opml.xml |
OPML list of the feeds |
/micropub, /micropub/media |
Micropub and media endpoints (Bearer token) |
/webmention |
Webmention receiver (POST answers 202; validated asynchronously) |
/websub |
WebSub hub for this site's own topics |
/.well-known/oauth-authorization-server, /auth/authorize, /auth/token, /auth/revoke |
IndieAuth server |
/auth/login, /auth/callback, /auth/logout |
Admin login via OIDC |
/admin/ |
Dashboard (notes, WebSub subscriber counts) |
/admin/new, /admin/edit/<id>, /admin/delete/<id> |
Note editor |
/admin/webmentions, /admin/webmentions/<id>, …/action, …/bulk-action |
Webmention moderation |
/admin/tokens, /admin/tokens/<id>/revoke |
Issued IndieAuth tokens |
/admin/syndication/<id>/retry |
Retry a failed crosspost |
/admin/metrics-dashboard, /admin/metrics, /admin/health |
Monitoring (HTML dashboard, JSON metrics, full diagnostics) |
/health |
Public liveness check ({"status": "ok", "version": …}); ?detailed=true requires login |
Background workers
Gunicorn runs four worker processes and create_app() runs in each, so each
process starts its own daemon threads. Work that must happen once per site is
coordinated through the data volume:
| Thread | Started when | Coordination | Work |
|---|---|---|---|
Webmention worker (webmention_worker.py) |
WEBMENTION_WORKER_ENABLED and not testing |
Non-blocking fcntl.flock on webmention-worker.lock next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in sending. |
Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries |
WebSub worker (websub_worker.py) |
WEBSUB_ENABLED and not testing |
Same pattern with websub-worker.lock |
Verifies subscription intents, delivers topic content (signed with X-Hub-Signature when the subscriber gave a secret), retries, expires leases; pings an external hub instead if WEBSUB_HUB_URL is set |
Syndication worker (syndication.py) |
At least one crosspost target configured, and not testing | No file lock. Rows in syndications are claimed with a 5-minute claimed_until lease by an atomic UPDATE, so any process may work a row but only one at a time; a version column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to threads_token.json. |
Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts |
Memory monitor (monitoring/memory.py) |
METRICS_ENABLED and not testing |
None (per-process metrics) | Samples process memory every METRICS_MEMORY_INTERVAL seconds |
All workers run inside an app context, are daemon threads (they end with the
process) and sleep for their poll interval when there is nothing to do. Under
the test suite (TESTING=True) none of them start; tests call the processing
functions directly.
Data storage
Content is kept in files and everything else in SQLite, both under
DATA_PATH (/data in the container):
- Notes: Markdown files at
notes/YYYY/MM/<slug>.md, written atomically (temporary file, then rename). The file holds the content; thenotesrow holds slug, path, timestamps, published flag, content hash and response properties (in_reply_to,like_of,repost_of,bookmark_of,content_format). Deletion is soft: the row getsdeleted_atand the file moves to.trash/YYYY/MM/, so Micropubundeletecan restore it. - Media:
media/YYYY/MM/<uuid>.<ext>plus<uuid>_<variant>.<ext>variants, staged inmedia/.tmp/while being processed. - Database:
starpunk.db(WAL mode): note metadata, tags, media records, sessions, IndieAuth codes and tokens, webmentions and the outgoing queue, syndications, WebSub subscriptions and queues, caches (author h-card, reply contexts), the FTS5 index andsite_state. See database.md. - Other files:
logs/starpunk.log(rotating), worker lock files,threads_token.json, anddebug/for failed uploads whenDEBUG_SAVE_FAILED_UPLOADS=true.
The FTS index is maintained by notes.py (SQLite cannot read the note files)
and rebuilt at start-up if it is empty.
Authentication model
There is one user, the site owner, identified by the ADMIN_ME URL. Details
are in simplified-auth-architecture.md.
- Admin login uses OpenID Connect (authorization code with PKCE S256 and a
nonce) against a configured provider (Pocket ID in production). Only the
subject
OIDC_ADMIN_SUBmay log in. The ID token's signature (via JWKS), issuer, audience and expiry are checked. Pending logins are kept inoidc_login_state. - Sessions: a random token in the
starpunk_sessioncookie (HttpOnly, SameSite=Lax, Secure whenSITE_URLis https), stored as a SHA-256 hash insessionsand valid forSESSION_LIFETIMEdays. No passwords are stored. - Micropub tokens: StarPunk is its own IndieAuth authorization server. A
client sends the owner to
/auth/authorize; the owner (logged in through OIDC) approves; the client exchanges the code at/auth/token(PKCE required). Codes and tokens are stored hashed. Tokens lastINDIEAUTH_TOKEN_LIFETIME_DAYSand can be revoked at/auth/revokeor in/admin/tokens. Scopes:create,update,delete,media,profile. - Development:
DEV_MODE=trueadds/dev/login, which logs in asDEV_ADMIN_MEwithout any checks. Never enabled in production; start-up logs a warning when it is on.
Configuration
All configuration comes from environment variables (optionally a .env
file), read once at start-up by starpunk/config.py. Validation fails fast:
SESSION_SECRET is always required; outside DEV_MODE, ADMIN_ME and the
four OIDC_* settings are required too. Crosspost targets switch on when
their credentials are set. The full list with defaults is in
deployment.md.
Caching
- Feed note list: per process, refreshed after
FEED_CACHE_SECONDSor as soon assite_state.content_versionchanges. Every change to a public note bumps that value in the database, so all gunicorn processes notice at once. - Rendered feeds:
feeds/cache.py, an in-memory LRU cache with a TTL (FEED_CACHE_MAX_SIZE,FEED_CACHE_SECONDS), keyed by a checksum of the notes pluscontent_version. Responses carryETagandCache-Control: public, max-age=…; conditional requests get304. - Author h-card (
author_profile, 24 hours) and reply contexts (reply_contexts) are cached in the database. - The webmention worker caches endpoint discovery results in memory.
Security posture
- Outbound requests (SSRF): fetches of URLs that come from outside
(webmention sources and targets, endpoint discovery, reply contexts, WebSub
callbacks, remote Micropub photos, IndieAuth client pages, crosspost link
cards) are checked first: every address the host resolves to must be public
(private, loopback, link-local, CGNAT, multicast and reserved ranges are
refused). The fetcher in
webmention_ssrf.py, used for webmentions, reply contexts, WebSub and remote Micropub photos, follows redirects by hand and re-checks each hop, connects to the address it checked (no second DNS lookup) and limits response size (1 MB by default) and total time. Crosspost link cards use their own check that also re-checks each redirect. The IndieAuth client-page fetch checks only the initial URL.DEV_MODErelaxes the checks for local testing. - HTML sanitising: rendered notes (Markdown or HTML sent by a Micropub
client) and webmention content are cleaned with nh3 allow-lists; search
highlighting escapes everything except
<mark>. - Rate limits:
/webmentionper client IP and per source domain (defaults 60 and 10 per 60 seconds);/websub30 requests per IP per minute, plus limits on subscriptions in total, per callback host and per callback address, and on requests waiting for verification. Request-rate counters are in memory and per process. - Client IP behind a proxy: with
TRUSTED_PROXY=truethe first address inX-Forwarded-Foris used as the client IP for rate limiting. Set it only when the app is reachable solely through a proxy that sets that header (Caddy does); without it, every client shares the proxy's address. - Auth: hashed session and token storage, single-use login state, PKCE for both OIDC and IndieAuth, redirect URIs checked against the client, post-login redirects limited to this site. Admin form POSTs rely on the SameSite=Lax session cookie (there are no per-form CSRF tokens).
- Uploads: request bodies are capped at 50 MB; images are processed by Pillow and stored under random (UUID) names.
- Container: runs as a non-root user. HTTPS and security headers (HSTS, CSP, X-Frame-Options and so on) come from the reverse proxy.
Observability
- Logs go to stdout and
logs/starpunk.log(10 MB × 10 files); each line carries the request's correlation ID. - Metrics (
monitoring/): HTTP request timings, database query timings (with a slow-query threshold), memory, pool statistics and business counters, in an in-memory buffer per process, shown at/admin/metrics-dashboardand/admin/metrics. /healthreports status, version and webmention-worker statistics;/health?detailed=truechecks database, filesystem and disk space;/admin/healthadds metrics, pool statistics and system information.
Last Updated: 2026-11-03 (v2.10.1)