Files
StarPunk/docs/architecture/overview.md
T
PhilandClaude Opus 5.5 75946be7b2 maint: Dependency upgrades and High backlog items (v2.11.0)
- All dependencies to current releases (97 known vulnerabilities -> 0),
  Python 3.14, uv pinned, requirements-test.txt, lxml/dateutil pinned
- CSRF tokens on admin forms, logout and IndieAuth consent
- IndieAuth client and crosspost link-card fetches via safe_get
- FLASK_DEBUG/FLASK_ENV default off/production; image sets them
- Foreign keys enforced on pooled connections; migration 018 removes orphans
- Rate limits shared across processes (rate_limits, migration 019)
- datetime.utcnow() and sqlite3 default adapters replaced
- compose.yaml fixed; duplicate dev_auth.py and test.ini removed

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 13:09:52 -07:00

22 KiB
Raw Permalink Blame History

StarPunk Architecture Overview

Version: 2.10.1 (2026-11-03) Status: Current. Describes the system as it is in the code today.

Purpose

StarPunk is a minimal, single-user IndieWeb blog/CMS. One person (the site owner) writes short notes, which are published as microformats2-marked HTML pages and as RSS, Atom and JSON feeds. The site takes part in the IndieWeb: it accepts posts from Micropub clients, issues its own IndieAuth tokens for them, sends and receives Webmentions, pushes feed updates with WebSub, and crossposts (POSSE) to GoToSocial, Bluesky and Threads.

Guiding principle: "every line of code must justify its existence". It is a single Flask application with server-side rendering, SQLite and plain files. There is no JavaScript build step, no external database and no message broker. Background work runs in threads inside the web processes.

Topic Document
Dependencies, runtime, tests, CI technology-stack.md
Database schema and migrations database.md
Build, configuration and production deployment deployment.md
Admin login (OIDC) and the built-in IndieAuth server simplified-auth-architecture.md
Micropub, IndieAuth, Webmention, microformats2, WebSub, feeds indieweb-protocols.md
Crossposting (POSSE) to GoToSocial, Bluesky, Threads syndication-architecture.md

Key decisions: ADR-064 (OIDC login + built-in IndieAuth server), ADR-063 (Webmention), ADR-065, ADR-066, ADR-069 and ADR-070 (crossposting), ADR-067 (Micropub update/delete), ADR-068 (likes and reposts), ADR-071 (WebSub), ADR-057 and ADR-058 (media), ADR-034 (search), ADR-052 (configuration), ADR-053 and ADR-060 (metrics), ADR-054 (logging), ADR-055 (errors).

System at a glance

                       Internet
                          │
                    ┌─────▼─────┐   TLS, security headers,
                    │   Caddy   │   sets X-Forwarded-For
                    └─────┬─────┘
                          │ HTTP :8000
┌─────────────────────────▼─────────────────────────────────────────┐
│ Container: gunicorn, 4 sync worker processes                       │
│                                                                    │
│ each process = Flask app (create_app) + daemon threads:            │
│   request handling    routes/* blueprints -> Jinja2 templates      │
│   webmention worker   works its queues only while holding a lock   │
│   websub worker       works its queues only while holding a lock   │
│   syndication worker  claims rows with a lease in SQLite           │
│   memory monitor      per-process metrics                          │
└───────────────┬──────────────────────────────┬─────────────────────┘
                │                              │
        ┌───────▼────────┐          ┌──────────▼──────────────────┐
        │ /data volume   │          │ Outbound HTTP               │
        │ starpunk.db    │          │ OIDC provider, IndieAuth    │
        │ notes/*.md     │          │ client pages, webmention    │
        │ media/         │          │ sources/targets, WebSub     │
        │ logs/, locks   │          │ callbacks, GoToSocial,      │
        └────────────────┘          │ Bluesky, Threads            │
                                    └─────────────────────────────┘

Request flow

  1. Caddy terminates TLS and proxies to gunicorn on port 8000.
  2. app.py builds the app once per worker process with starpunk.create_app() (starpunk/__init__.py). That function loads and validates configuration, configures logging, cleans up old debug and temporary upload files, creates the schema and runs migrations (under a lock, see database.md), creates the connection pool, installs the HTTP metrics middleware, configures the feed cache and rate limiters, fills the FTS index if it is empty, registers blueprints, error handlers and template helpers, and starts the background threads.
  3. before_request assigns a correlation ID (included in every log line) and, if a starpunk_session cookie is present, verifies it and sets g.me.
  4. The blueprint handler runs. Database access goes through get_db(), which hands out a pooled, request-scoped SQLite connection.
  5. after_request hooks add Link: </webmention>; rel="webmention" to HTML responses, WebSub Link rel=hub / rel=self headers to topic responses, and request metrics with an X-Request-ID header.
  6. Errors are rendered by the central handlers in starpunk/errors.py (JSON for Micropub, HTML pages from templates/<code>.html elsewhere).

Writing a note (admin form or Micropub) goes through starpunk/notes.py, which writes the Markdown file and the notes row together. It then updates the FTS index, queues outgoing webmentions, fetches a reply context for replies, marks crossposts pending, and tells WebSub what changed (bumping the site's content_version and queueing deliveries). The HTTP response does not wait for outbound network calls; the workers make them later.

Components

Application core (starpunk/)

Module Responsibility
__init__.py App factory create_app(), logging setup, request hooks, /health, worker start-up, __version__
config.py Reads environment variables (and .env) into app.config; validates and fails fast
notes.py Note CRUD with file and database kept in sync; soft delete and undelete; triggers follow-up work
models.py Note and other data classes; Markdown rendering and HTML sanitising (nh3 allow-list)
utils.py Slugs, note file paths (notes/YYYY/MM/slug.md), atomic writes, trash, date formatting, absolute_url_for
slug_utils.py Custom slug (mp-slug) validation and timestamp fallback slugs
markdown_ext.py Markdown extension that turns bare URLs into links
media.py Image upload and validation (including HEIC), resizing, variants (thumb/small/medium/large), attachment to notes
tags.py Tag normalisation and note–tag links
search.py FTS5 full-text search (falls back to LIKE if FTS5 is unavailable); index maintained by application code
auth.py Admin login with OpenID Connect (PKCE, nonce, ID-token checks with PyJWT); sessions; require_auth
indieauth.py Built-in IndieAuth authorization server: client discovery, codes, hashed access tokens, scopes
dev_auth.py Development-only login (DEV_MODE=true)
micropub.py Micropub create, update, delete, undelete; queries (q=config, source, syndicate-to, category); photos
author_discovery.py Fetches and caches the owner's h-card from ADMIN_ME (24-hour cache)
mf2_utils.py Helpers for consuming microformats2: representative h-card, responding entry, response type, author
reply_context.py Fetches and caches the post a reply responds to
webmention_sender.py Extracts links from rendered notes and queues outgoing webmentions
webmention_processor.py Validates incoming webmentions: fetch source, parse mf2, classify, sanitise content
webmention_worker.py Background thread working the incoming and outgoing webmention queues
webmention_queries.py Webmention queries for display and moderation
webmention_ssrf.py SSRF-safe fetcher: public-address checks, pinned connections, manual redirects, size and time limits
websub.py WebSub publisher (discovery headers and links, content_version, queueing) and hub request handling
websub_worker.py Background thread: verification of intent, content delivery, publish pings to an external hub
syndication.py POSSE clients (Mastodon API/GoToSocial, Bluesky, Threads), reconcile logic, syndication worker
rate_limiter.py In-memory sliding-window rate limiter
errors.py Central error handlers
feeds/ rss.py (feedgen), atom.py, json_feed.py, opml.py, negotiation.py (Accept header), cache.py (LRU + TTL feed cache with ETags)
feed.py Backward-compatibility re-export of feeds.rss
monitoring/ Metrics buffer, HTTP middleware, database timing wrapper, memory monitor, business metrics
database/ schema.py (INITIAL_SCHEMA_SQL), init.py (init_db), pool.py (connection pool)
migrations.py Migration runner for migrations/NNN_*.sql

Routes (starpunk/routes/)

Blueprint File Paths
public public.py /, /note/<slug>, /tag/<tag>, /likes, /media/<path>, feeds, /opml.xml
auth auth.py /auth/login (GET, POST), /auth/callback, /auth/logout (POST)
indieauth indieauth.py /.well-known/oauth-authorization-server, /auth/authorize (GET, POST), /auth/authorize/approve, /auth/token (GET, POST), /auth/revoke
micropub micropub.py /micropub (GET, POST), /micropub/media (POST), /media (POST, legacy alias)
admin admin.py /admin/… (see below)
search search.py /search, /api/search
webmention webmention.py /webmention (GET, POST); registered only when WEBMENTION_ENABLED
websub websub.py /websub (GET for information, POST to subscribe/unsubscribe)
dev_auth dev_auth.py /dev/login; registered only when DEV_MODE=true

/health is defined directly in create_app().

Templates and static files

Jinja2 templates live in templates/: public pages, admin/, partials/ (for example responses.html for webmentions and reaction.html for likes, reposts and bookmarks) and one page per HTTP error. templates/base.html carries the discovery links (rel=micropub, indieauth-metadata, authorization_endpoint, token_endpoint, webmention, hub/self, me, feeds). Static assets are one stylesheet (static/css/style.css) and one script (static/js/preview.js, the Markdown preview in the admin editor). Everything works without JavaScript.

Public URL surface

Path Purpose
/ Home page: recent published notes (h-feed); likes are kept off it
/note/<slug> Note permalink (h-entry), with approved webmentions and reply context
/tag/<tag> Notes with a tag
/likes Likes
/search, /api/search Full-text search page and JSON API
/media/<path> Uploaded images and their variants
/feed Feed chosen by the Accept header (RSS by default; Vary: Accept)
/feed.rss, /feed.atom, /feed.json RSS 2.0, Atom 1.0, JSON Feed 1.1
/feed.xml Legacy alias of /feed.rss
/opml.xml OPML list of the feeds
/micropub, /micropub/media Micropub and media endpoints (Bearer token)
/webmention Webmention receiver (POST answers 202; validated asynchronously)
/websub WebSub hub for this site's own topics
/.well-known/oauth-authorization-server, /auth/authorize, /auth/token, /auth/revoke IndieAuth server
/auth/login, /auth/callback, /auth/logout Admin login via OIDC
/admin/ Dashboard (notes, WebSub subscriber counts)
/admin/new, /admin/edit/<id>, /admin/delete/<id> Note editor
/admin/webmentions, /admin/webmentions/<id>, …/action, …/bulk-action Webmention moderation
/admin/tokens, /admin/tokens/<id>/revoke Issued IndieAuth tokens
/admin/syndication/<id>/retry Retry a failed crosspost
/admin/metrics-dashboard, /admin/metrics, /admin/health Monitoring (HTML dashboard, JSON metrics, full diagnostics)
/health Public liveness check ({"status": "ok", "version": …}); ?detailed=true requires login

Background workers

Gunicorn runs four worker processes and create_app() runs in each, so each process starts its own daemon threads. Work that must happen once per site is coordinated through the data volume:

Thread Started when Coordination Work
Webmention worker (webmention_worker.py) WEBMENTION_WORKER_ENABLED and not testing Non-blocking fcntl.flock on webmention-worker.lock next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in sending. Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries
WebSub worker (websub_worker.py) WEBSUB_ENABLED and not testing Same pattern with websub-worker.lock Verifies subscription intents, delivers topic content (signed with X-Hub-Signature when the subscriber gave a secret), retries, expires leases; pings an external hub instead if WEBSUB_HUB_URL is set
Syndication worker (syndication.py) At least one crosspost target configured, and not testing No file lock. Rows in syndications are claimed with a 5-minute claimed_until lease by an atomic UPDATE, so any process may work a row but only one at a time; a version column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to threads_token.json. Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts
Memory monitor (monitoring/memory.py) METRICS_ENABLED and not testing None (per-process metrics) Samples process memory every METRICS_MEMORY_INTERVAL seconds

All workers run inside an app context, are daemon threads (they end with the process) and sleep for their poll interval when there is nothing to do. Under the test suite (TESTING=True) none of them start; tests call the processing functions directly.

Data storage

Content is kept in files and everything else in SQLite, both under DATA_PATH (/data in the container):

  • Notes: Markdown files at notes/YYYY/MM/<slug>.md, written atomically (temporary file, then rename). The file holds the content; the notes row holds slug, path, timestamps, published flag, content hash and response properties (in_reply_to, like_of, repost_of, bookmark_of, content_format). Deletion is soft: the row gets deleted_at and the file moves to .trash/YYYY/MM/, so Micropub undelete can restore it.
  • Media: media/YYYY/MM/<uuid>.<ext> plus <uuid>_<variant>.<ext> variants, staged in media/.tmp/ while being processed.
  • Database: starpunk.db (WAL mode): note metadata, tags, media records, sessions, IndieAuth codes and tokens, webmentions and the outgoing queue, syndications, WebSub subscriptions and queues, caches (author h-card, reply contexts), the FTS5 index and site_state. See database.md.
  • Other files: logs/starpunk.log (rotating), worker lock files, threads_token.json, and debug/ for failed uploads when DEBUG_SAVE_FAILED_UPLOADS=true.

The FTS index is maintained by notes.py (SQLite cannot read the note files) and rebuilt at start-up if it is empty.

Authentication model

There is one user, the site owner, identified by the ADMIN_ME URL. Details are in simplified-auth-architecture.md.

  • Admin login uses OpenID Connect (authorization code with PKCE S256 and a nonce) against a configured provider (Pocket ID in production). Only the subject OIDC_ADMIN_SUB may log in. The ID token's signature (via JWKS), issuer, audience and expiry are checked. Pending logins are kept in oidc_login_state.
  • Sessions: a random token in the starpunk_session cookie (HttpOnly, SameSite=Lax, Secure when SITE_URL is https), stored as a SHA-256 hash in sessions and valid for SESSION_LIFETIME days. No passwords are stored.
  • Micropub tokens: StarPunk is its own IndieAuth authorization server. A client sends the owner to /auth/authorize; the owner (logged in through OIDC) approves; the client exchanges the code at /auth/token (PKCE required). Codes and tokens are stored hashed. Tokens last INDIEAUTH_TOKEN_LIFETIME_DAYS and can be revoked at /auth/revoke or in /admin/tokens. Scopes: create, update, delete, media, profile.
  • Development: DEV_MODE=true adds /dev/login, which logs in as DEV_ADMIN_ME without any checks. Never enabled in production; start-up logs a warning when it is on.

Configuration

All configuration comes from environment variables (optionally a .env file), read once at start-up by starpunk/config.py. Validation fails fast: SESSION_SECRET is always required; outside DEV_MODE, ADMIN_ME and the four OIDC_* settings are required too. Crosspost targets switch on when their credentials are set. The full list with defaults is in deployment.md.

Caching

  • Feed note list: per process, refreshed after FEED_CACHE_SECONDS or as soon as site_state.content_version changes. Every change to a public note bumps that value in the database, so all gunicorn processes notice at once.
  • Rendered feeds: feeds/cache.py, an in-memory LRU cache with a TTL (FEED_CACHE_MAX_SIZE, FEED_CACHE_SECONDS), keyed by a checksum of the notes plus content_version. Responses carry ETag and Cache-Control: public, max-age=…; conditional requests get 304.
  • Author h-card (author_profile, 24 hours) and reply contexts (reply_contexts) are cached in the database.
  • The webmention worker caches endpoint discovery results in memory.

Security posture

  • Outbound requests (SSRF): fetches of URLs that come from outside (webmention sources and targets, endpoint discovery, reply contexts, WebSub callbacks, remote Micropub photos, IndieAuth client pages, crosspost link cards) are checked first: every address the host resolves to must be public (private, loopback, link-local, CGNAT, multicast and reserved ranges are refused). Every fetch of a user-supplied URL (webmentions, reply contexts, WebSub, remote Micropub photos, IndieAuth client pages and, since v2.11.0, crosspost link cards) uses the fetcher in webmention_ssrf.py: it follows redirects by hand and re-checks each hop, connects to the address it checked (no second DNS lookup) and limits response size (1 MB by default) and total time. DEV_MODE relaxes the checks for local testing.
  • HTML sanitising: rendered notes (Markdown or HTML sent by a Micropub client) and webmention content are cleaned with nh3 allow-lists; search highlighting escapes everything except <mark>.
  • Rate limits: /webmention per client IP and per source domain (defaults 60 and 10 per 60 seconds); /websub 30 requests per IP per minute, plus limits on subscriptions in total, per callback host and per callback address, and on requests waiting for verification. Request-rate counters live in the rate_limits table, shared by all server processes (v2.11.0).
  • Client IP behind a proxy: with TRUSTED_PROXY=true the first address in X-Forwarded-For is used as the client IP for rate limiting. Set it only when the app is reachable solely through a proxy that sets that header (Caddy does); without it, every client shares the proxy's address.
  • Auth: hashed session and token storage, single-use login state, PKCE for both OIDC and IndieAuth, redirect URIs checked against the client, post-login redirects limited to this site. Admin form POSTs, logout and the IndieAuth consent carry a CSRF token: an HMAC of the session cookie, checked by starpunk/csrf.py (v2.11.0), on top of the SameSite=Lax cookie.
  • Uploads: request bodies are capped at 50 MB; images are processed by Pillow and stored under random (UUID) names.
  • Container: runs as a non-root user. HTTPS and security headers (HSTS, CSP, X-Frame-Options and so on) come from the reverse proxy.

Observability

  • Logs go to stdout and logs/starpunk.log (10 MB × 10 files); each line carries the request's correlation ID.
  • Metrics (monitoring/): HTTP request timings, database query timings (with a slow-query threshold), memory, pool statistics and business counters, in an in-memory buffer per process, shown at /admin/metrics-dashboard and /admin/metrics.
  • /health reports status, version and webmention-worker statistics; /health?detailed=true checks database, filesystem and disk space; /admin/health adds metrics, pool statistics and system information.

Last Updated: 2026-11-03 (v2.10.1)