Files
StarPunk/docs/architecture/indieweb-protocols.md
T
PhilandClaude Opus 5.5 0716984958 docs: Bring all documentation up to date for v2.10.1
- architecture/: overview, technology stack, auth (OIDC + built-in
  IndieAuth), syndication (POSSE) rewritten; new database.md,
  deployment.md and indieweb-protocols.md; legacy stack marked historical
- decisions/: reconstructed ADR-060 and ADR-063; 31 status corrections
  (superseded auth ADRs, proposed -> accepted); INDEX rebuilt
- design/: reconstructed the v1.6.0 documents the webmention code cites;
  reports for 18 v2 releases without a design folder; INDEX rebuilt
- projectplan/: BACKLOG rewritten (shipped items removed, open issues from
  the reviews added); RELEASE.md for v2.0.0-v2.10.1; INDEX
- standards/: development setup (uv, tests, DEV_MODE), testing (CI test
  stage), git and release flow, versioning, coding standards, cookies
- README, CLAUDE.md, .env.example (every setting the code reads, with
  defaults) brought up to date
- 137 broken links repaired (old absolute paths, moved files); stale ADR
  references in code comments corrected (comments only)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:40:19 -07:00

25 KiB

IndieWeb and W3C Protocols

How StarPunk implements each protocol it speaks, as of v2.10.1, with its compliance status and the gaps that remain. Each section links to the decision records and review documents behind it.

Protocol Role Status Main code
Micropub Server (posts and media) Reviewed against the Recommendation in v2.9.0; deliberate deviations listed starpunk/micropub.py, starpunk/routes/micropub.py
IndieAuth Authorization server for Micropub clients Authorization code + PKCE, metadata, revocation starpunk/indieauth.py, starpunk/routes/indieauth.py
Webmention Sender and receiver Every MUST and SHOULD checked in v2.7.0 and v2.7.2; known limits listed starpunk/webmention_*.py, starpunk/routes/webmention.py
Microformats2 Publisher and consumer Reviewed in v2.8.0 templates/, starpunk/mf2_utils.py, starpunk/author_discovery.py
WebSub Publisher and built-in hub (own topics only) Adversarial review in v2.10.1 starpunk/websub.py, starpunk/websub_worker.py, starpunk/routes/websub.py
Feeds RSS 2.0, Atom 1.0, JSON Feed 1.1, OPML 2.0 Stable starpunk/feeds/, starpunk/routes/public.py

Crossposting to other networks (POSSE) is covered in syndication-architecture.md.

Shared infrastructure: the SSRF-safe fetcher

Webmention (fetching sources, discovering endpoints, sending), WebSub (verification and delivery) and Micropub (downloading remote photos) all make requests to URLs chosen by someone else. They use starpunk/webmention_ssrf.py:

  • validate_url_for_fetch(): http(s) only. Every address from getaddrinfo must be public. Private, loopback, link-local, multicast, reserved, unspecified, CGNAT (100.64.0.0/10), 0.0.0.0/8 and IPv4-mapped forms are blocked. It returns the checked address, preferring IPv4. DEV_MODE allows localhost.
  • DNS pinning (v2.7.2): each request connects to the checked IP, with the original host in the Host header and as the TLS SNI name. The certificate is still verified against the real hostname, so a rebinding DNS server can't send the second lookup elsewhere.
  • safe_get() follows up to 10 redirects by hand and checks each hop. The response is capped at 1 MB by default.
  • safe_post() doesn't follow redirects.
  • Hard deadlines (v2.10.1): the whole exchange, not each read, is limited to 10 s. When the deadline passes, the client is closed, so a server trickling bytes can't hold a worker.
  • DNS resolution failures raise ResolutionError (a ValueError), which callers can treat as temporary.
  • User-Agent: StarPunk/<version> Webmention (+<SITE_URL>).

Workers that drain queues (Webmention, WebSub) run as one daemon thread per gunicorn process. Each takes a non-blocking flock on a lock file in the data directory, so only one process works the queue at a time and another takes over if it dies.


Micropub

Spec: https://www.w3.org/TR/micropub/. Decisions: ADR-028, ADR-029, ADR-057 (photos), ADR-067 (update/delete), ADR-068 (likes/reposts). Review: v2.9.0 Micropub review.

Endpoints and discovery

Endpoint Purpose
GET /micropub?q=... Queries
POST /micropub Create, update, delete, undelete
POST /micropub/media (and legacy POST /media) Media endpoint

Every page has <link rel="micropub"> plus the IndieAuth discovery links (see IndieAuth). Authentication, token placement, scopes and error codes are described in simplified-auth-architecture.md §3.

Create

Accepts form-encoded, multipart (with photo files) and JSON bodies. Only h=entry / type: ["h-entry"] is accepted; other types get 400. Success is 201 with Location set to the permalink /note/<slug>.

Property / command Handling
content String, or {"html": ...} / {"text": ...}. Text is preferred. HTML-only content is stored with content_format = 'html' (migration 015) and rendered through render_markdown() (Markdown, then nh3 allowlist sanitization)
summary Used as the text when there is no content
name Stored as a # name heading on the first line (StarPunk has no title column)
category Tags (strings only; person tags as h-card objects are ignored)
in-reply-to, like-of, repost-of, bookmark-of URL or h-cite. At most one of like/repost/bookmark. Content is optional for these
photo URLs, {"value", "alt"} objects, or uploaded files; at most 4 (ADR-057), duplicates collapsed. Own /media/ URLs are looked up; other URLs are downloaded through safe_get (20 MB cap). Any photo that can't be resolved means 400 and nothing is created
published ISO 8601; converted to naive UTC; invalid gets 400
post-status published (default) or draft; others get 400
visibility public; private and unlisted are saved as drafts (StarPunk has no private posts); others get 400
video, audio 400 (not supported)
location, checkin, others Ignored (allowed by 3.3.5)
mp-slug Custom slug
mp-syndicate-to Crosspost targets. Absent means none (v2.9.0). See syndication-architecture.md

mp-* keys are read as commands from the top level or from inside JSON properties, and are never stored. JSON property values must be arrays; a bare scalar is wrapped on create. If anything fails after the note is created, the note is removed and 500 is returned, so the client never believes a half-made post failed.

Post types advertised in q=config: note, article, photo, reply, like, repost, bookmark.

Update, delete, undelete

  • Update (JSON only, update scope) targets a /note/<slug> URL on this site.
    • replace works on content, name, category, in-reply-to, post-status and photo.
    • add works on category and photo only.
    • delete takes a list of property names, or values to remove for category and photo.
    • Values must be arrays. An update without any operation gets 400. Unsupported properties get 400.
    • Photos are resolved before anything changes. Success is 204.
    • Edits follow crossposts and are re-announced to Webmention targets and WebSub subscribers.
  • Delete (delete scope) is soft: the note is moved to the trash, its page returns 410, and undelete restores it. Both return 204.

Queries

Query Scope Result
q=config any token media-endpoint, syndicate-to, q list, post-types
q=syndicate-to any token {"syndicate-to": [{"uid", "name"}]}
q=source&url=...[&properties[]=...] create/update/delete/draft One post as mf2 JSON. Content comes back as {"html": ...} if it was authored as HTML. Includes syndication URLs and post-status
q=source without url same {"items": [...]}, drafts included, limit (default 10, max 50) and offset
q=category[&filter=...] same Tags in use, most used first

Unknown or missing q gets 400.

Media endpoint

POST /micropub/media, multipart with one part named file, media or create scope. It returns 201 with Location: <SITE_URL>/media/<path>. Images are JPEG, PNG, GIF, WebP or HEIC, up to 50 MB in. They are optimized and get responsive variants (starpunk/media.py, ADR-058). Validation failures get 400.

Known deviations and gaps

  • 403 for insufficient_scope (RFC 6750) where the Micropub error table says 401. Kept on purpose.
  • add is only supported for category and photo. published, syndication and location can't be updated. mp-syndicate-to is only read on create.
  • location / checkin are ignored, and person tags are dropped.
  • video and audio are refused.
  • The access_token query parameter is still accepted on GET.
  • Only notes (/note/<slug>) can be the target of update/delete/source.

IndieAuth

StarPunk is its own IndieAuth authorization server for Micropub clients, issuing tokens for the single identity ADMIN_ME. The admin approves clients on a consent screen protected by OIDC login. Supported: the authorization code flow with mandatory PKCE S256, iss in responses (RFC 9207), the metadata endpoint, a profile-only flow, a token endpoint, RFC 7009 revocation, and legacy token verification (GET /auth/token). Client redirect_uris must be same-origin or published by the client (JSON metadata or rel=redirect_uri).

Full description, scopes and gaps: simplified-auth-architecture.md. Decision: ADR-064.

Known gaps: scopes_supported doesn't list draft. There are no introspection, userinfo or refresh tokens. The client metadata fetch doesn't use the pinned SSRF fetcher.


Webmention

Spec: https://www.w3.org/TR/webmention/ (Recommendation, 12 January 2017). Decision: ADR-063. Reviews: v2.6.9 fixes, v2.7.0 compliance, v2.7.2 remaining SHOULDs.

Endpoint discovery (advertising)

With WEBMENTION_ENABLED (default true), every HTML response carries Link: </webmention>; rel="webmention", and base.html has <link rel="webmention" href="/webmention">. When it is off, the receiving route isn't registered at all.

Receiving

POST /webmention (routes/webmention.py):

  1. Rate limits: 60 requests per minute per client IP and 10 per minute per source domain (WEBMENTION_RATE_LIMIT_*). Over the limit gets 429 with Retry-After. The limiter is in memory, per process. The client IP comes from X-Forwarded-For only when TRUSTED_PROXY=true.
  2. Form-encoded (JSON also accepted). source and target are required, http(s), at most 2048 characters, and different from each other. Otherwise 400.
  3. The target's host and port must be this site's. The fragment is stripped before storing. The target must be a published note: anything else, including the home page, gets 400 invalid_target.
  4. The row is upserted into webmentions (unique on source + target) as pending, and 202 is returned. A repeat from the same source resets it to pending (an update, 3.2.4).

Verification is asynchronous (WebmentionWorker, poll WEBMENTION_POLL_INTERVAL 10 s, batch of 5):

  • Fetch the source with safe_get (redirects followed and checked, 1 MB, 10 s), with an Accept header preferring HTML.
  • Exact-match verification (3.2.2), per media type:
    • HTML: every link-bearing attribute (lxml iterlinks: a/link href, img/video/audio src, ...), resolved against the final URL.
    • JSON: any string value.
    • Other text/*: substring match.
    • Any other type: no match.
    • Fragments are ignored on both sides.
  • Parse mf2 (HTML only) with mf2py:
    • mf2_utils.responding_entry() picks the h-entry that responds to the target.
    • response_type() classifies it as reply, like, repost, bookmark or mention.
    • entry_author() follows the authorship algorithm without extra fetches: author property, then the parent h-feed's author, then rel=author, then the representative h-card.
    • Also stored: a content snippet (e-content or p-summary, sanitized with nh3, at most 500 characters), the source's u-url and dt-published (migration 014).
    • The author URL and photo are kept only if they are absolute http(s) URLs (XSS, 4.1).
  • Outcomes:
    • Valid: validation_status = 'valid'. A first-time mention gets moderation_status = 'approved' if WEBMENTION_AUTO_APPROVE (default true), otherwise pending. An update keeps the admin's earlier moderation decision.
    • Source returns 4xx (including 410), or no longer links: invalid, which hides it (3.2.4 deletion).
    • Network error, timeout or 5xx: a mention that was valid before stays valid with its old data. A new one becomes error.
    • A source that is one of the note's own syndicated copies (syndications.remote_url) is invalid (v2.8.0).
  • Re-send on display change (3.1.4, v2.7.2): if what the note shows for this webmention changed, the note's own outgoing webmentions are re-queued. That covers a new response, a changed author or snippet, or a change of shown/hidden. Unchanged re-validations send nothing, so two sites doing the same don't ping-pong.

Moderation and display. /admin/webmentions lists, filters, and approves, hides or deletes mentions, one at a time or in bulk. Changes that alter what is shown also re-send the note's webmentions. Note pages show approved, valid mentions grouped as replies (u-comment h-cite), likes, reposts and bookmarks (facepiles), and mentions. See Microformats2.

Sending

Queued by webmention_sender.queue_webmentions_for_note() from notes.py. Sending isn't gated by WEBMENTION_ENABLED; it runs whenever the worker does (WEBMENTION_WORKER_ENABLED).

Event Targets
Publish Reply / like / repost / bookmark target first, then <a href> links from the rendered HTML in document order (not inside <pre>/<code>), absolute http(s) only, de-duplicated, at most 10
Edit (published) The same, plus every target notified before and links the edit removed (not capped) (3.1.4)
Delete Every previously notified target is re-queued (3.1.5); for a hard delete the queue rows are detached first so the cascade doesn't remove them
Undelete Re-queued again
Displayed responses change Re-queued (see above)

Re-queuing resets the cached endpoint, so discovery runs again (3.1.4).

Endpoint discovery (3.1.2): GET the target with safe_get (redirects checked). The first Link header with a rel token webmention wins. The header is parsed per RFC 8288: several links per header, several headers, rel token lists, case-insensitive. Otherwise the first <link> or <a> with rel~=webmention and an href, in document order. Relative URLs resolve against the final URL, an empty href means the target itself, and the query string is kept.

Send (3.1.3): a form-encoded POST through safe_post, pinned and with no redirects followed (a 301/302 would turn the POST into a GET).

Result Outcome
Any 2xx sent
429, 5xx, network error Retried after 1 h, 6 h, 24 h; failed after the 4th attempt
Other 3xx/4xx, SSRF-blocked endpoint failed
No endpoint no_endpoint

Deleted posts: /note/<slug> returns 410 Gone with a tombstone page for soft-deleted notes, and for hard-deleted notes that had sent webmentions (3.1.5).

Security (section 4)

  • XSS: author URL and photo must be http(s). Snippets are sanitized with nh3 (and re-sanitized after truncation). The mentions list uses striptags.
  • SSRF: all fetches and POSTs go through the pinned fetcher described above, so the endpoint URL is checked before POSTing.
  • DoS: rate limits on receiving, a 1 MB / 10 s cap on fetches, and asynchronous processing.

Known limits

From the v2.7.0 review, still current:

  • Home-page mentions (and mentions of anything that isn't a published note) are refused rather than stored.
  • Discovery ignores HTTP cache headers (5.3). Targets are fetched only when a note is created, edited or deleted.
  • The 410 page is a generic tombstone without microformats.
  • error rows (transient failures of a new mention, or unexpected errors) are not retried automatically. They are re-checked only when the sender re-sends.
  • The 202 response has no Location status URL.

Microformats2

Specs: microformats2 parsing, h-entry, h-card, h-feed, h-cite, representative h-card, authorship, Post Type Discovery. Decisions: ADR-040, ADR-061. Review: v2.8.0 microformats2 review.

What pages publish

Page Markup
/ (home) h-feed with p-name, hidden u-url, hidden p-author h-card (photo, name, URL, note), feed u-photo; each note an h-entry preview with u-uid
/note/<slug> h-entry: p-name only when the note starts with a # Heading (so plain notes have no implied name); u-photo as direct children above e-content; u-url u-uid with dt-published (RFC 3339 UTC Z); dt-updated; p-category with rel=tag; u-syndication links; hidden p-author h-card
Replies u-in-reply-to h-cite holding the cached reply context (p-content, u-url)
Likes / reposts / bookmarks u-like-of / u-repost-of / u-bookmark-of link with "★ Liked" / "🔁 Reposted" / "🔖 Bookmarked"
Received responses u-comment h-cite (replies: p-author h-card with u-url on an inner link, u-photo, p-name; e-content; u-url; dt-published), and u-like, u-repost, u-bookmark, u-mention h-cites. Faces beyond 8 are hidden, not omitted
/likes, /tag/<tag> h-feed with p-name, u-url, p-author h-card

Post Type Discovery gives note, article, reply, like, repost, bookmark and photo correctly.

base.html always emits rel=me and rel=author pointing at the author's home page (author.url, else author.me, else ADMIN_ME), even on a fresh install (v2.8.0). Further rel=me links come from the discovered profile. Because the p-author h-card's u-url matches the rel=me URL, parsers treat it as the page's representative h-card. Without a discovered profile there is no h-card, only the rel links.

Consuming

  • Author profile (author_discovery.py, ADR-061): on login, StarPunk fetches ADMIN_ME, picks its representative h-card with mf2_utils.representative_hcard() (uid and url match the page, then url matches a rel=me, then a single h-card whose url is the page), parsed with the URL after redirects. It stores name, photo, URL, note and rel-me links, cached for 24 hours. If discovery fails, the stale cache is used.
  • Webmention sources: responding_entry, response_type and entry_author (see Webmention).
  • Reply context (reply_context.py): title, snippet, author and site name of the post being replied to, liked, reposted or bookmarked, cached in reply_contexts.

Known gaps

From the v2.8.0 review, "Not done":

  • The reply-context h-cite doesn't carry the cached author h-card, name or published time.
  • Photo alt text falls back to "Image" when a photo-only post has no caption.
  • Person tags (p-category h-card) aren't supported.
  • Webmentions received before v2.8.0 keep NULL source details until they are re-validated.

WebSub

Spec: https://www.w3.org/TR/websub/. Decision: ADR-071, including its v2.10.1 amendments, which replace the original text where they differ.

Topics

All absolute URLs under SITE_URL:

Topic Content type Updated when
/ text/html A public non-like note changes
/feed.atom, /feed.rss, /feed.json feed types A public non-like note changes
/likes text/html A public like changes
/tag/<name> text/html A note with that tag (before or after the change) changes

/feed.xml (legacy) advertises /feed.rss as its rel=self. The negotiated /feed advertises the topic of the format it served.

Discovery

Topic responses (200/304) carry Link: <hub>; rel="hub" and Link: <topic>; rel="self". The same links are embedded: <link rel="hub"> and <link rel="self"> in HTML heads, a hub <link> in Atom, an <atom:link rel="hub"> in RSS, and "hubs" in JSON Feed. All of this happens only while WEBSUB_ENABLED (default true).

On change

websub.note_changed() runs on create, edit, publish/unpublish, delete and undelete, and once photos are attached. If the note is or was public, it:

  1. bumps content_version, which is in the feed cache key and the ETag, so every process serves fresh feeds at once;
  2. queues one delivery per active subscription for each affected topic. Only the latest content is sent: a newer delivery replaces a waiting one.

Drafts that stay drafts change nothing.

Hub

POST /websub, form-encoded only (case-insensitive check):

  • hub.mode is subscribe or unsubscribe.

  • hub.topic must be one of the topics above, after decoding unreserved percent-encodings and comparing scheme and host case-insensitively. Tag topics must name an existing tag.

  • hub.callback must:

    • be printable, at most 2,048 characters, and absolute http(s);
    • not be on this site;
    • use port 80, 443 or 1024 and up;
    • pass the SSRF check.

    It is stored with an IDNA host.

  • hub.secret must be non-empty and under 200 bytes. hub.lease_seconds is clamped to 1 to 30 days (default 10). Both are ignored for unsubscriptions.

  • The hub answers 202, then verifies later. Errors are 4xx with a plain-text reason.

Limits (429 when hit):

  • 30 requests per minute per client IP (per process; needs TRUSTED_PROXY=true behind a proxy);
  • at most 200 pending verifications;
  • WEBSUB_MAX_SUBSCRIPTIONS (1000) in total;
  • WEBSUB_MAX_PER_HOST (25) per callback IP address.

The limits count pending requests too. Renewals are always allowed.

WebSubWorker (file lock, poll WEBSUB_POLL_INTERVAL 5 s) handles each queued item on its own, so one bad item can't stop it:

  • Verification of intent (5.3): GET the callback (existing query kept) with hub.mode, the topic exactly as requested, hub.challenge and hub.lease_seconds. Success is a 2xx whose body equals the challenge. Replies over 4 KB and redirects fail. Network/DNS errors and 5xx are retried after 1, 5 and 15 minutes. A request id per intent (migration 017) stops an older verification from consuming a newer request. A subscription row exists only once verified.
  • Distribution (7): the topic is rendered inside the app, once per pass, and POSTed with its Content-Type, Link rel=hub/self, and X-Hub-Signature: sha256=<HMAC> when there is a secret.
    • 2xx: success.
    • 410: removes the subscription.
    • Other failures: retried after 1 min, 5 min, 30 min, 2 h and 6 h, then that delivery is dropped. The subscription stays until its lease ends.
    • A topic that renders 404/410 ends its subscriptions with a hub.mode=denied notice.
  • Expired subscriptions are pruned hourly.

Outbound requests use safe_get / safe_post: pinned, no redirects, 10 s hard deadline.

External hub: with WEBSUB_HUB_URL set, StarPunk advertises that hub and sends it a hub.mode=publish ping per changed topic. Its own /websub refuses new subscriptions but still accepts unsubscriptions.

The admin dashboard shows subscribers per topic and deliveries being retried.

Known limits

  • The hub serves only this site's topics, by design. websub.rocks' hub tests use their own topics, so they can't be run against it.
  • Rate limiting is per process and in memory.
  • Deliveries always carry the full topic content (no diffs). Fat pings aren't sent to external hubs.

Feeds

Decisions: ADR-014, ADR-038, ADR-059.

URL Format
/feed.rss RSS 2.0 (with atom:link rel=self and Media RSS)
/feed.xml Legacy alias of /feed.rss
/feed.atom Atom 1.0
/feed.json JSON Feed 1.1
/feed Content negotiation on Accept (RSS default; 406 with X-Available-Formats if nothing fits; Vary: Accept)
/opml.xml OPML 2.0 listing the three feeds
  • Feeds contain the newest FEED_MAX_ITEMS (50) published notes, with full HTML content and media. Likes are excluded, as on the home page. Reposts and bookmarks start with a "🔁 Reposted" / "🔖 Bookmarked" line.
  • Feeds are cached for FEED_CACHE_SECONDS (300). The cache key and ETag include content_version, so any change to a public note invalidates them in every process. If-None-Match gets 304.
  • base.html advertises the negotiated /feed (as RSS) and the OPML file with rel=alternate. Atom and JSON Feed are listed in the OPML but have no <link rel=alternate> of their own.
  • WebSub hub links are embedded in each feed (see above).

Gaps: there are no per-tag feeds (tag pages are h-feeds and WebSub topics, but have no RSS/Atom/JSON equivalent). The /likes page has no feed.