Files
StarPunk/docs/architecture/overview.md
T
PhilandClaude Opus 5.5 84411a16d9 feat: MCP server for AI clients (v2.15.0, ADR-075)
Add an MCP server at SITE_URL/mcp so AI assistants (claude.ai connectors,
Claude Code) can search, read and write notes.

- starpunk/mcp_server: official MCP SDK 2.3, stateless Streamable HTTP with
  JSON responses, served by uvicorn as a sidecar from the same image.
  Ten tools; writes go through the Micropub handlers.
- IndieAuth: RFC 8707 resource indicators bound to codes and tokens
  (migration 022), client ID metadata document + public client metadata,
  draft in scopes_supported, default MCP scopes, consent screen note.
- BACKGROUND_WORKERS_ENABLED flag so the sidecar runs no workers.
- Docs: ADR-075, architecture/mcp.md and updates; CHANGELOG.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 16:36:58 -07:00

24 KiB
Raw Blame History

StarPunk Architecture Overview

Version: 2.14.0 (2026-11-10) Status: Current. Describes the system as it is in the code today.

Purpose

StarPunk is a minimal, single-user IndieWeb blog/CMS. One person (the site owner) writes short notes, which are published as microformats2-marked HTML pages and as RSS, Atom and JSON feeds. The site takes part in the IndieWeb: it accepts posts from Micropub clients, issues its own IndieAuth tokens for them, sends and receives Webmentions, pushes feed updates with WebSub, and crossposts (POSSE) to GoToSocial, Bluesky and Threads. Responses to those crossposts are pulled back and shown on the note (backfeed), and the owner can be notified of new responses.

Guiding principle: "every line of code must justify its existence". It is a single Flask application with server-side rendering, SQLite and plain files. There is no JavaScript build step, no external database and no message broker. Background work runs in threads inside the web processes.

Topic Document
Dependencies, runtime, tests, CI technology-stack.md
Database schema and migrations database.md
Build, configuration and production deployment deployment.md
Admin login (OIDC) and the built-in IndieAuth server simplified-auth-architecture.md
Micropub, IndieAuth, Webmention, microformats2, WebSub, feeds indieweb-protocols.md
Crossposting (POSSE) to GoToSocial, Bluesky, Threads syndication-architecture.md

Key decisions: ADR-064 (OIDC login + built-in IndieAuth server), ADR-063 (Webmention), ADR-065, ADR-066, ADR-069 and ADR-070 (crossposting), ADR-067 (Micropub update/delete), ADR-068 (likes and reposts), ADR-071 (WebSub), ADR-072 (metrics dashboard removed), ADR-073 (backfeed), ADR-074 (notifications), ADR-057 and ADR-058 (media), ADR-034 (search), ADR-052 (configuration), ADR-054 (logging), ADR-055 (errors).

System at a glance

                       Internet
                          │
                    ┌─────▼─────┐   TLS, security headers,
                    │   Caddy   │   sets X-Forwarded-For
                    └─────┬─────┘
                          │ HTTP :8000
┌─────────────────────────▼─────────────────────────────────────────┐
│ Container: gunicorn, 4 sync worker processes                       │
│                                                                    │
│ each process = Flask app (create_app) + daemon threads:            │
│   request handling    routes/* blueprints -> Jinja2 templates      │
│   webmention worker   works its queues only while holding a lock   │
│   websub worker       works its queues only while holding a lock   │
│   syndication worker  claims rows with a lease in SQLite           │
└───────────────┬──────────────────────────────┬─────────────────────┘
                │                              │
        ┌───────▼────────┐          ┌──────────▼──────────────────┐
        │ /data volume   │          │ Outbound HTTP               │
        │ starpunk.db    │          │ OIDC provider, IndieAuth    │
        │ notes/*.md     │          │ client pages, webmention    │
        │ media/         │          │ sources/targets, WebSub     │
        │ avatars/       │          │ callbacks, GoToSocial,      │
        │ logs/, locks   │          │ Bluesky, Threads, avatars,  │
        └────────────────┘          │ notifications (Apprise)     │
                                    └─────────────────────────────┘

Request flow

  1. Caddy terminates TLS and proxies to gunicorn on port 8000.
  2. app.py builds the app once per worker process with starpunk.create_app() (starpunk/__init__.py). That function loads and validates configuration, configures logging, cleans up old debug and temporary upload files, creates the schema and runs migrations (under a lock, see database.md), creates the connection pool, configures the feed cache and rate limiters, fills the FTS index if it is empty, registers blueprints, error handlers and template helpers, and starts the background threads.
  3. before_request assigns a correlation ID (included in every log line) and, if a starpunk_session cookie is present, verifies it and sets g.me.
  4. The blueprint handler runs. Database access goes through get_db(), which hands out a pooled, request-scoped SQLite connection.
  5. after_request hooks add Link: </webmention>; rel="webmention" to HTML responses, WebSub Link rel=hub / rel=self headers to topic responses.
  6. Errors are rendered by the central handlers in starpunk/errors.py (JSON for Micropub, HTML pages from templates/<code>.html elsewhere).

Writing a note (admin form or Micropub) goes through starpunk/notes.py, which writes the Markdown file and the notes row together. It then updates the FTS index, queues outgoing webmentions, fetches a reply context for replies, marks crossposts pending, and tells WebSub what changed (bumping the site's content_version and queueing deliveries). The HTTP response does not wait for outbound network calls; the workers make them later.

Components

Application core (starpunk/)

Module Responsibility
__init__.py App factory create_app(), logging setup, request hooks, /health, worker start-up, __version__
config.py Reads environment variables (and .env) into app.config; validates and fails fast
notes.py Note CRUD with file and database kept in sync; soft delete and undelete; triggers follow-up work
models.py Note and other data classes; Markdown rendering and HTML sanitising (nh3 allow-list)
utils.py Slugs, note file paths (notes/YYYY/MM/slug.md), atomic writes, trash, date formatting, absolute_url_for
slug_utils.py Custom slug (mp-slug) validation and timestamp fallback slugs
markdown_ext.py Markdown extension that turns bare URLs into links
media.py Image upload and validation (including HEIC), resizing, variants (thumb/small/medium/large), attachment to notes
tags.py Tag normalisation and note–tag links
search.py FTS5 full-text search (falls back to LIKE if FTS5 is unavailable); index maintained by application code
auth.py Admin login with OpenID Connect (PKCE, nonce, ID-token checks with PyJWT); sessions; require_auth
indieauth.py Built-in IndieAuth authorization server: client discovery, codes, hashed access tokens, scopes
dev_auth.py Development-only login (DEV_MODE=true)
micropub.py Micropub create, update, delete, undelete; queries (q=config, source, syndicate-to, category); photos
author_discovery.py Fetches and caches the owner's h-card from ADMIN_ME (24-hour cache)
mf2_utils.py Helpers for consuming microformats2: representative h-card, responding entry, response type, author
reply_context.py Fetches and caches the post a reply responds to
webmention_sender.py Extracts links from rendered notes and queues outgoing webmentions
webmention_processor.py Validates incoming webmentions: fetch source, parse mf2, classify, sanitise content
webmention_worker.py Background thread working the incoming and outgoing webmention queues; also runs avatar upkeep and the notification sweep
webmention_queries.py Webmention queries for display and moderation
webmention_ssrf.py SSRF-safe fetcher: public-address checks, pinned connections, manual redirects, size and time limits
websub.py WebSub publisher (discovery headers and links, content_version, queueing) and hub request handling
websub_worker.py Background thread: verification of intent, content delivery, publish pings to an external hub
syndication.py POSSE clients (Mastodon API/GoToSocial, Bluesky, Threads), reconcile logic, syndication worker (its loop also calls backfeed.process_due())
backfeed.py Backfeed: reads responses to crossposts from the silos, stores them as webmentions, schedules checks and backfills
avatars.py Saves response authors' avatars as 96 px WebP files in avatars/; fills missing ones and removes unused ones
notifications.py Notifications through Apprise: the sweep of new responses and immediate problem messages
csrf.py CSRF tokens (HMAC of the session cookie) for admin forms, logout and the IndieAuth consent
rate_limiter.py In-memory sliding-window rate limiter
errors.py Central error handlers
feeds/ rss.py (feedgen), atom.py, json_feed.py, opml.py, negotiation.py (Accept header), cache.py (LRU + TTL feed cache with ETags)
feed.py Backward-compatibility re-export of feeds.rss
database/ schema.py (INITIAL_SCHEMA_SQL), init.py (init_db), pool.py (connection pool)
migrations.py Migration runner for migrations/NNN_*.sql

Routes (starpunk/routes/)

Blueprint File Paths
public public.py /, /note/<slug>, /tag/<tag>, /likes, /media/<path>, /avatars/<name>, feeds, /opml.xml
auth auth.py /auth/login (GET, POST), /auth/callback, /auth/logout (POST)
indieauth indieauth.py /.well-known/oauth-authorization-server, /auth/authorize (GET, POST), /auth/authorize/approve, /auth/token (GET, POST), /auth/revoke
micropub micropub.py /micropub (GET, POST), /micropub/media (POST), /media (POST, legacy alias)
admin admin.py /admin/… (see below)
search search.py /search, /api/search
webmention webmention.py /webmention (GET, POST); registered only when WEBMENTION_ENABLED
websub websub.py /websub (GET for information, POST to subscribe/unsubscribe)
dev_auth dev_auth.py /dev/login; registered only when DEV_MODE=true

/health is defined directly in create_app().

Templates and static files

Jinja2 templates live in templates/: public pages, admin/, partials/ (for example responses.html for webmentions and reaction.html for likes, reposts and bookmarks) and one page per HTTP error. templates/base.html carries the discovery links (rel=micropub, indieauth-metadata, authorization_endpoint, token_endpoint, webmention, hub/self, me, feeds). Static assets are one stylesheet (static/css/style.css); admin pages have small inline scripts. Everything works without JavaScript.

Public URL surface

Path Purpose
/ Home page: recent published notes (h-feed); likes are kept off it
/note/<slug> Note permalink (h-entry), with approved responses (webmentions and backfed silo responses), a counts line with web actions, and reply context
/tag/<tag> Notes with a tag
/likes Likes
/search, /api/search Full-text search page and JSON API
/media/<path> Uploaded images and their variants
/avatars/<name> Saved copies of response authors' avatars
/feed Feed chosen by the Accept header (RSS by default; Vary: Accept)
/feed.rss, /feed.atom, /feed.json RSS 2.0, Atom 1.0, JSON Feed 1.1
/feed.xml Legacy alias of /feed.rss
/opml.xml OPML list of the feeds
/micropub, /micropub/media Micropub and media endpoints (Bearer token)
/webmention Webmention receiver (POST answers 202; validated asynchronously)
/websub WebSub hub for this site's own topics
/.well-known/oauth-authorization-server, /auth/authorize, /auth/token, /auth/revoke IndieAuth server
/auth/login, /auth/callback, /auth/logout Admin login via OIDC
/admin/ Dashboard (notes, WebSub subscriber counts)
/admin/new, /admin/edit/<id>, /admin/delete/<id> Note editor
/admin/webmentions, /admin/webmentions/<id>, …/action, …/bulk-action Webmention moderation
/admin/tokens, /admin/tokens/<id>/revoke Issued IndieAuth tokens
/admin/syndication/<id>/retry Retry a failed crosspost
/admin/syndication/<id>/check (POST) Check one crosspost for responses now
/admin/backfeed/backfill (POST) Check every crosspost for responses once (backfill)
/admin/notifications/test (POST) Send a test notification
/admin/health Full diagnostics (JSON): database, pool statistics, filesystem, disk
/health Public liveness check ({"status": "ok", "version": …}); ?detailed=true requires login
/dev/login Development login; only when DEV_MODE=true

/, /tag/<tag> and /likes show each note's response counts, linking to its responses (v2.13.4).

Background workers

Gunicorn runs four worker processes and create_app() runs in each, so each process starts its own daemon threads. Work that must happen once per site is coordinated through the data volume:

Thread Started when Coordination Work
Webmention worker (webmention_worker.py) WEBMENTION_WORKER_ENABLED and not testing Non-blocking fcntl.flock on webmention-worker.lock next to the database. Only the holder works the queues; the others try again each poll interval. The OS releases the lock if the holder dies, and the new holder resets rows stuck in sending. Validates received webmentions; discovers endpoints and sends queued outgoing webmentions, with retries; saves missing response avatars every 10 minutes and removes unused ones weekly (v2.13.0); sweeps new responses into notifications and sets webmentions.notified_at (v2.14.0, notifications.py, ADR-074)
WebSub worker (websub_worker.py) WEBSUB_ENABLED and not testing Same pattern with websub-worker.lock Verifies subscription intents, delivers topic content (signed with X-Hub-Signature when the subscriber gave a secret), retries, expires leases; pings an external hub instead if WEBSUB_HUB_URL is set
Syndication worker (syndication.py) At least one crosspost target configured, and not testing No file lock. Rows in syndications are claimed with a 5-minute claimed_until lease by an atomic UPDATE, so any process may work a row but only one at a time; a version column keeps edits made during a send. Threads token refreshes are serialised with a lock file next to threads_token.json. Creates, updates or deletes the remote post for each (note, target) pair; up to 8 attempts. Then, only when BACKFEED_ENABLED, backfeed: checks copies for responses, each claimed with a backfeed_claimed_until lease (v2.13.0, backfeed.py, ADR-073)

BACKGROUND_WORKERS_ENABLED=false starts none of them; the MCP sidecar (below) sets it so the workers only run in the main container.

All workers run inside an app context, are daemon threads (they end with the process) and sleep for their poll interval when there is nothing to do. Under the test suite (TESTING=True) none of them start; tests call the processing functions directly.

Data storage

Content is kept in files and everything else in SQLite, both under DATA_PATH (/data in the container):

  • Notes: Markdown files at notes/YYYY/MM/<slug>.md, written atomically (temporary file, then rename). The file holds the content; the notes row holds slug, path, timestamps, published flag, content hash and response properties (in_reply_to, like_of, repost_of, bookmark_of, content_format). Deletion is soft: the row gets deleted_at and the file moves to .trash/YYYY/MM/, so Micropub undelete can restore it.
  • Media: media/YYYY/MM/<uuid>.<ext> plus <uuid>_<variant>.<ext> variants, staged in media/.tmp/ while being processed.
  • Database: starpunk.db (WAL mode): note metadata, tags, media records, sessions, IndieAuth codes and tokens, webmentions (including backfed silo responses, marked by origin and remote_id) and the outgoing queue, syndications (with backfeed state and silo-only counts), WebSub subscriptions and queues, caches (author h-card, reply contexts), the FTS5 index and site_state. See database.md.
  • Avatars: avatars/<sha256>.webp, saved copies of response authors' avatars (v2.13.0); they can be re-fetched, so losing them is harmless.
  • Other files: logs/starpunk.log (rotating), worker lock files, threads_token.json, and debug/ for failed uploads when DEBUG_SAVE_FAILED_UPLOADS=true.

The FTS index is maintained by notes.py (SQLite cannot read the note files) and rebuilt at start-up if it is empty.

Authentication model

There is one user, the site owner, identified by the ADMIN_ME URL. Details are in simplified-auth-architecture.md.

  • Admin login uses OpenID Connect (authorization code with PKCE S256 and a nonce) against a configured provider (Pocket ID in production). Only the subject OIDC_ADMIN_SUB may log in. The ID token's signature (via JWKS), issuer, audience and expiry are checked. Pending logins are kept in oidc_login_state.
  • Sessions: a random token in the starpunk_session cookie (HttpOnly, SameSite=Lax, Secure when SITE_URL is https), stored as a SHA-256 hash in sessions and valid for SESSION_LIFETIME days. No passwords are stored.
  • Micropub tokens: StarPunk is its own IndieAuth authorization server. A client sends the owner to /auth/authorize; the owner (logged in through OIDC) approves; the client exchanges the code at /auth/token (PKCE required). Codes and tokens are stored hashed. Tokens last INDIEAUTH_TOKEN_LIFETIME_DAYS and can be revoked at /auth/revoke or in /admin/tokens. Scopes: create, update, delete, media, draft, profile.
  • MCP (v2.15.0, ADR-075): AI clients use the same flow, identified by a client ID metadata document and asking for resource=SITE_URL/mcp. Their tokens are bound to that resource and checked by the MCP sidecar, a second container (uvicorn, official MCP SDK) that the proxy sends /mcp to. Its tools read, search and write notes through the Micropub handlers. See mcp.md.
  • Development: DEV_MODE=true adds /dev/login, which logs in as DEV_ADMIN_ME without any checks. Never enabled in production; start-up logs a warning when it is on.

Configuration

All configuration comes from environment variables (optionally a .env file), read once at start-up by starpunk/config.py. Validation fails fast: SESSION_SECRET is always required; outside DEV_MODE, ADMIN_ME and the four OIDC_* settings are required too. Crosspost targets switch on when their credentials are set. The full list with defaults is in deployment.md.

Caching

  • Feed note list: per process, refreshed after FEED_CACHE_SECONDS or as soon as site_state.content_version changes. Every change to a public note bumps that value in the database, so all gunicorn processes notice at once.
  • Rendered feeds: feeds/cache.py, an in-memory LRU cache with a TTL (FEED_CACHE_MAX_SIZE, FEED_CACHE_SECONDS), keyed by a checksum of the notes plus content_version. Responses carry ETag and Cache-Control: public, max-age=…; conditional requests get 304.
  • Author h-card (author_profile, 24 hours) and reply contexts (reply_contexts) are cached in the database.
  • The webmention worker caches endpoint discovery results in memory.
  • Response avatars are saved once as files named by the SHA-256 of the source URL and served from /avatars/<name> with Cache-Control: public, max-age=604800 (one week).

Security posture

  • Outbound requests (SSRF): every fetch of a URL that comes from outside (webmention sources and targets, endpoint discovery, reply contexts, WebSub callbacks, remote Micropub photos, IndieAuth client pages, crosspost link cards since v2.11.0, response avatars since v2.13.0) goes through the fetcher in webmention_ssrf.py. Every address the host resolves to must be public (private, loopback, link-local, CGNAT, multicast and reserved ranges are refused); it follows redirects by hand and re-checks each hop, connects to the address it checked (no second DNS lookup) and limits response size (1 MB by default) and total time. DEV_MODE relaxes the checks for local testing. Exceptions: avatars on the site's own GOTOSOCIAL_URL host are fetched directly, as the crossposting client does, since behind the same proxy it may resolve to an internal address (v2.13.1); Bluesky CDN avatars are fetched from the account's PDS with com.atproto.sync.getBlob, itself through the safe fetcher. Apprise notifications go to the owner-configured NOTIFY_URLS, outside the safe fetcher.
  • HTML sanitising: rendered notes (Markdown or HTML sent by a Micropub client) and webmention content are cleaned with nh3 allow-lists; search highlighting escapes everything except <mark>.
  • Rate limits: /webmention per client IP and per source domain (defaults 60 and 10 per 60 seconds); /websub 30 requests per IP per minute, plus limits on subscriptions in total, per callback host and per callback address, and on requests waiting for verification. Request-rate counters live in the rate_limits table, shared by all server processes (v2.11.0).
  • Client IP behind a proxy: with TRUSTED_PROXY=true the first address in X-Forwarded-For is used as the client IP for rate limiting. Set it only when the app is reachable solely through a proxy that sets that header (Caddy does); without it, every client shares the proxy's address.
  • Auth: hashed session and token storage, single-use login state, PKCE for both OIDC and IndieAuth, redirect URIs checked against the client, post-login redirects limited to this site. Admin form POSTs, logout and the IndieAuth consent carry a CSRF token: an HMAC of the session cookie, checked by starpunk/csrf.py (v2.11.0), on top of the SameSite=Lax cookie.
  • Uploads: request bodies are capped at 50 MB; images are processed by Pillow and stored under random (UUID) names.
  • Container: runs as a non-root user. HTTPS and security headers (HSTS, CSP, X-Frame-Options and so on) come from the reverse proxy.

Observability

  • Logs go to stdout and logs/starpunk.log (10 MB × 10 files); each line carries the request's correlation ID.
  • /health reports status, version and webmention-worker statistics; /health?detailed=true checks database, filesystem and disk space; /admin/health adds pool statistics and filesystem details. (The metrics dashboard was removed in v2.12.0, see ADR-072.)

Last Updated: 2026-11-10 (v2.14.0)