- All dependencies to current releases (97 known vulnerabilities -> 0), Python 3.14, uv pinned, requirements-test.txt, lxml/dateutil pinned - CSRF tokens on admin forms, logout and IndieAuth consent - IndieAuth client and crosspost link-card fetches via safe_get - FLASK_DEBUG/FLASK_ENV default off/production; image sets them - Foreign keys enforced on pooled connections; migration 018 removes orphans - Rate limits shared across processes (rate_limits, migration 019) - datetime.utcnow() and sqlite3 default adapters replaced - compose.yaml fixed; duplicate dev_auth.py and test.ini removed Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
25 KiB
IndieWeb and W3C Protocols
How StarPunk implements each protocol it speaks, as of v2.10.1, with its compliance status and the gaps that remain. Each section links to the decision records and review documents behind it.
| Protocol | Role | Status | Main code |
|---|---|---|---|
| Micropub | Server (posts and media) | Reviewed against the Recommendation in v2.9.0; deliberate deviations listed | starpunk/micropub.py, starpunk/routes/micropub.py |
| IndieAuth | Authorization server for Micropub clients | Authorization code + PKCE, metadata, revocation | starpunk/indieauth.py, starpunk/routes/indieauth.py |
| Webmention | Sender and receiver | Every MUST and SHOULD checked in v2.7.0 and v2.7.2; known limits listed | starpunk/webmention_*.py, starpunk/routes/webmention.py |
| Microformats2 | Publisher and consumer | Reviewed in v2.8.0 | templates/, starpunk/mf2_utils.py, starpunk/author_discovery.py |
| WebSub | Publisher and built-in hub (own topics only) | Adversarial review in v2.10.1 | starpunk/websub.py, starpunk/websub_worker.py, starpunk/routes/websub.py |
| Feeds | RSS 2.0, Atom 1.0, JSON Feed 1.1, OPML 2.0 | Stable | starpunk/feeds/, starpunk/routes/public.py |
Crossposting to other networks (POSSE) is covered in syndication-architecture.md.
Shared infrastructure: the SSRF-safe fetcher
Webmention (fetching sources, discovering endpoints, sending), WebSub
(verification and delivery) and Micropub (downloading remote photos) all
make requests to URLs chosen by someone else. They use
starpunk/webmention_ssrf.py:
validate_url_for_fetch(): http(s) only. Every address fromgetaddrinfomust be public. Private, loopback, link-local, multicast, reserved, unspecified, CGNAT (100.64.0.0/10),0.0.0.0/8and IPv4-mapped forms are blocked. It returns the checked address, preferring IPv4.DEV_MODEallows localhost.- DNS pinning (v2.7.2): each request connects to the checked IP, with
the original host in the
Hostheader and as the TLS SNI name. The certificate is still verified against the real hostname, so a rebinding DNS server can't send the second lookup elsewhere. safe_get()follows up to 10 redirects by hand and checks each hop. The response is capped at 1 MB by default.safe_post()doesn't follow redirects.- Hard deadlines (v2.10.1): the whole exchange, not each read, is limited to 10 s. When the deadline passes, the client is closed, so a server trickling bytes can't hold a worker.
- DNS resolution failures raise
ResolutionError(aValueError), which callers can treat as temporary. - User-Agent:
StarPunk/<version> Webmention (+<SITE_URL>).
Workers that drain queues (Webmention, WebSub) run as one daemon thread per
gunicorn process. Each takes a non-blocking flock on a lock file in the
data directory, so only one process works the queue at a time and another
takes over if it dies.
Micropub
Spec: https://www.w3.org/TR/micropub/. Decisions: ADR-028, ADR-029, ADR-057 (photos), ADR-067 (update/delete), ADR-068 (likes/reposts). Review: v2.9.0 Micropub review.
Endpoints and discovery
| Endpoint | Purpose |
|---|---|
GET /micropub?q=... |
Queries |
POST /micropub |
Create, update, delete, undelete |
POST /micropub/media (and legacy POST /media) |
Media endpoint |
Every page has <link rel="micropub"> plus the IndieAuth discovery links
(see IndieAuth). Authentication, token placement, scopes and
error codes are described in
simplified-auth-architecture.md §3.
Create
Accepts form-encoded, multipart (with photo files) and JSON bodies. Only
h=entry / type: ["h-entry"] is accepted; other types get 400. Success is
201 with Location set to the permalink /note/<slug>.
| Property / command | Handling |
|---|---|
content |
String, or {"html": ...} / {"text": ...}. Text is preferred. HTML-only content is stored with content_format = 'html' (migration 015) and rendered through render_markdown() (Markdown, then nh3 allowlist sanitization) |
summary |
Used as the text when there is no content |
name |
Stored as a # name heading on the first line (StarPunk has no title column) |
category |
Tags (strings only; person tags as h-card objects are ignored) |
in-reply-to, like-of, repost-of, bookmark-of |
URL or h-cite. At most one of like/repost/bookmark. Content is optional for these |
photo |
URLs, {"value", "alt"} objects, or uploaded files; at most 4 (ADR-057), duplicates collapsed. Own /media/ URLs are looked up; other URLs are downloaded through safe_get (20 MB cap). Any photo that can't be resolved means 400 and nothing is created |
published |
ISO 8601; converted to naive UTC; invalid gets 400 |
post-status |
published (default) or draft; others get 400 |
visibility |
public; private and unlisted are saved as drafts (StarPunk has no private posts); others get 400 |
video, audio |
400 (not supported) |
location, checkin, others |
Ignored (allowed by 3.3.5) |
mp-slug |
Custom slug |
mp-syndicate-to |
Crosspost targets. Absent means none (v2.9.0). See syndication-architecture.md |
mp-* keys are read as commands from the top level or from inside JSON
properties, and are never stored. JSON property values must be arrays; a
bare scalar is wrapped on create. If anything fails after the note is
created, the note is removed and 500 is returned, so the client never
believes a half-made post failed.
Post types advertised in q=config: note, article, photo, reply, like,
repost, bookmark.
Update, delete, undelete
- Update (JSON only,
updatescope) targets a/note/<slug>URL on this site.replaceworks oncontent,name,category,in-reply-to,post-statusandphoto.addworks oncategoryandphotoonly.deletetakes a list of property names, or values to remove forcategoryandphoto.- Values must be arrays. An update without any operation gets 400. Unsupported properties get 400.
- Photos are resolved before anything changes. Success is 204.
- Edits follow crossposts and are re-announced to Webmention targets and WebSub subscribers.
- Delete (
deletescope) is soft: the note is moved to the trash, its page returns 410, andundeleterestores it. Both return 204.
Queries
| Query | Scope | Result |
|---|---|---|
q=config |
any token | media-endpoint, syndicate-to, q list, post-types |
q=syndicate-to |
any token | {"syndicate-to": [{"uid", "name"}]} |
q=source&url=...[&properties[]=...] |
create/update/delete/draft | One post as mf2 JSON. Content comes back as {"html": ...} if it was authored as HTML. Includes syndication URLs and post-status |
q=source without url |
same | {"items": [...]}, drafts included, limit (default 10, max 50) and offset |
q=category[&filter=...] |
same | Tags in use, most used first |
Unknown or missing q gets 400.
Media endpoint
POST /micropub/media, multipart with one part named file, media or
create scope. It returns 201 with Location: <SITE_URL>/media/<path>.
Images are JPEG, PNG, GIF, WebP or HEIC, up to 50 MB in. They are optimized
and get responsive variants (starpunk/media.py,
ADR-058). Validation
failures get 400.
Known deviations and gaps
- 403 for
insufficient_scope(RFC 6750) where the Micropub error table says 401. Kept on purpose. addis only supported forcategoryandphoto.published,syndicationandlocationcan't be updated.mp-syndicate-tois only read on create.location/checkinare ignored, and person tags are dropped.videoandaudioare refused.- The
access_tokenquery parameter is still accepted on GET. - Only notes (
/note/<slug>) can be the target of update/delete/source.
IndieAuth
StarPunk is its own IndieAuth authorization server for Micropub clients,
issuing tokens for the single identity ADMIN_ME. The admin approves
clients on a consent screen protected by OIDC login. Supported: the
authorization code flow with mandatory PKCE S256, iss in responses (RFC
9207), the metadata endpoint, a profile-only flow, a token endpoint, RFC
7009 revocation, and legacy token verification (GET /auth/token). Client
redirect_uris must be same-origin or published by the client (JSON
metadata or rel=redirect_uri).
Full description, scopes and gaps: simplified-auth-architecture.md. Decision: ADR-064.
Known gaps: scopes_supported doesn't list draft. There are no
introspection, userinfo or refresh tokens. The client metadata fetch
doesn't use the pinned SSRF fetcher.
Webmention
Spec: https://www.w3.org/TR/webmention/ (Recommendation, 12 January 2017). Decision: ADR-063. Reviews: v2.6.9 fixes, v2.7.0 compliance, v2.7.2 remaining SHOULDs.
Endpoint discovery (advertising)
With WEBMENTION_ENABLED (default true), every HTML response carries
Link: </webmention>; rel="webmention", and base.html has
<link rel="webmention" href="/webmention">. When it is off, the receiving
route isn't registered at all.
Receiving
POST /webmention (routes/webmention.py):
- Rate limits: 60 requests per minute per client IP and 10 per minute per
source domain (
WEBMENTION_RATE_LIMIT_*). Over the limit gets 429 withRetry-After. Counts are shared by all server processes (rate_limitstable, v2.11.0). The client IP comes fromX-Forwarded-Foronly whenTRUSTED_PROXY=true. - Form-encoded (JSON also accepted).
sourceandtargetare required, http(s), at most 2048 characters, and different from each other. Otherwise 400. - The target's host and port must be this site's. The fragment is stripped
before storing. The target must be a published note: anything else,
including the home page, gets 400
invalid_target. - The row is upserted into
webmentions(unique on source + target) aspending, and 202 is returned. A repeat from the same source resets it topending(an update, 3.2.4).
Verification is asynchronous (WebmentionWorker, poll
WEBMENTION_POLL_INTERVAL 10 s, batch of 5):
- Fetch the source with
safe_get(redirects followed and checked, 1 MB, 10 s), with anAcceptheader preferring HTML. - Exact-match verification (3.2.2), per media type:
- HTML: every link-bearing attribute (
lxml iterlinks:a/linkhref,img/video/audiosrc, ...), resolved against the final URL. - JSON: any string value.
- Other
text/*: substring match. - Any other type: no match.
- Fragments are ignored on both sides.
- HTML: every link-bearing attribute (
- Parse mf2 (HTML only) with mf2py:
mf2_utils.responding_entry()picks the h-entry that responds to the target.response_type()classifies it as reply, like, repost, bookmark or mention.entry_author()follows the authorship algorithm without extra fetches: author property, then the parent h-feed's author, thenrel=author, then the representative h-card.- Also stored: a content snippet (e-content or p-summary, sanitized with
nh3, at most 500 characters), the source's
u-urlanddt-published(migration 014). - The author URL and photo are kept only if they are absolute http(s) URLs (XSS, 4.1).
- Outcomes:
- Valid:
validation_status = 'valid'. A first-time mention getsmoderation_status = 'approved'ifWEBMENTION_AUTO_APPROVE(default true), otherwisepending. An update keeps the admin's earlier moderation decision. - Source returns 4xx (including 410), or no longer links:
invalid, which hides it (3.2.4 deletion). - Network error, timeout or 5xx: a mention that was valid before stays
valid with its old data. A new one becomes
error. - A source that is one of the note's own syndicated copies
(
syndications.remote_url) isinvalid(v2.8.0).
- Valid:
- Re-send on display change (3.1.4, v2.7.2): if what the note shows for this webmention changed, the note's own outgoing webmentions are re-queued. That covers a new response, a changed author or snippet, or a change of shown/hidden. Unchanged re-validations send nothing, so two sites doing the same don't ping-pong.
Moderation and display. /admin/webmentions lists, filters, and
approves, hides or deletes mentions, one at a time or in bulk. Changes that
alter what is shown also re-send the note's webmentions. Note pages show
approved, valid mentions grouped as replies (u-comment h-cite), likes,
reposts and bookmarks (facepiles), and mentions. See
Microformats2.
Sending
Queued by webmention_sender.queue_webmentions_for_note() from notes.py.
Sending isn't gated by WEBMENTION_ENABLED; it runs whenever the worker
does (WEBMENTION_WORKER_ENABLED).
| Event | Targets |
|---|---|
| Publish | Reply / like / repost / bookmark target first, then <a href> links from the rendered HTML in document order (not inside <pre>/<code>), absolute http(s) only, de-duplicated, at most 10 |
| Edit (published) | The same, plus every target notified before and links the edit removed (not capped) (3.1.4) |
| Delete | Every previously notified target is re-queued (3.1.5); for a hard delete the queue rows are detached first so the cascade doesn't remove them |
| Undelete | Re-queued again |
| Displayed responses change | Re-queued (see above) |
Re-queuing resets the cached endpoint, so discovery runs again (3.1.4).
Endpoint discovery (3.1.2): GET the target with safe_get (redirects
checked). The first Link header with a rel token webmention wins. The
header is parsed per RFC 8288: several links per header, several headers,
rel token lists, case-insensitive. Otherwise the first <link> or <a>
with rel~=webmention and an href, in document order. Relative URLs
resolve against the final URL, an empty href means the target itself, and
the query string is kept.
Send (3.1.3): a form-encoded POST through safe_post, pinned and with
no redirects followed (a 301/302 would turn the POST into a GET).
| Result | Outcome |
|---|---|
| Any 2xx | sent |
| 429, 5xx, network error | Retried after 1 h, 6 h, 24 h; failed after the 4th attempt |
| Other 3xx/4xx, SSRF-blocked endpoint | failed |
| No endpoint | no_endpoint |
Deleted posts: /note/<slug> returns 410 Gone with a tombstone page
for soft-deleted notes, and for hard-deleted notes that had sent
webmentions (3.1.5).
Security (section 4)
- XSS: author URL and photo must be http(s). Snippets are sanitized with
nh3 (and re-sanitized after truncation). The mentions list uses
striptags. - SSRF: all fetches and POSTs go through the pinned fetcher described above, so the endpoint URL is checked before POSTing.
- DoS: rate limits on receiving, a 1 MB / 10 s cap on fetches, and asynchronous processing.
Known limits
From the v2.7.0 review, still current:
- Home-page mentions (and mentions of anything that isn't a published note) are refused rather than stored.
- Discovery ignores HTTP cache headers (5.3). Targets are fetched only when a note is created, edited or deleted.
- The 410 page is a generic tombstone without microformats.
errorrows (transient failures of a new mention, or unexpected errors) are not retried automatically. They are re-checked only when the sender re-sends.- The 202 response has no
Locationstatus URL.
Microformats2
Specs: microformats2 parsing, h-entry, h-card, h-feed, h-cite, representative h-card, authorship, Post Type Discovery. Decisions: ADR-040, ADR-061. Review: v2.8.0 microformats2 review.
What pages publish
| Page | Markup |
|---|---|
/ (home) |
h-feed with p-name, hidden u-url, hidden p-author h-card (photo, name, URL, note), feed u-photo; each note an h-entry preview with u-uid |
/note/<slug> |
h-entry: p-name only when the note starts with a # Heading (so plain notes have no implied name); u-photo as direct children above e-content; u-url u-uid with dt-published (RFC 3339 UTC Z); dt-updated; p-category with rel=tag; u-syndication links; hidden p-author h-card |
| Replies | u-in-reply-to h-cite holding the cached reply context (p-content, u-url) |
| Likes / reposts / bookmarks | u-like-of / u-repost-of / u-bookmark-of link with "★ Liked" / "🔁 Reposted" / "🔖 Bookmarked" |
| Received responses | u-comment h-cite (replies: p-author h-card with u-url on an inner link, u-photo, p-name; e-content; u-url; dt-published), and u-like, u-repost, u-bookmark, u-mention h-cites. Faces beyond 8 are hidden, not omitted |
/likes, /tag/<tag> |
h-feed with p-name, u-url, p-author h-card |
Post Type Discovery gives note, article, reply, like, repost, bookmark and photo correctly.
Identity links
base.html always emits rel=me and rel=author pointing at the author's
home page (author.url, else author.me, else ADMIN_ME), even on a fresh
install (v2.8.0). Further rel=me links come from the discovered profile.
Because the p-author h-card's u-url matches the rel=me URL, parsers
treat it as the page's representative h-card. Without a discovered profile
there is no h-card, only the rel links.
Consuming
- Author profile (
author_discovery.py, ADR-061): on login, StarPunk fetchesADMIN_ME, picks its representative h-card withmf2_utils.representative_hcard()(uid and url match the page, then url matches arel=me, then a single h-card whose url is the page), parsed with the URL after redirects. It stores name, photo, URL, note and rel-me links, cached for 24 hours. If discovery fails, the stale cache is used. - Webmention sources:
responding_entry,response_typeandentry_author(see Webmention). - Reply context (
reply_context.py): title, snippet, author and site name of the post being replied to, liked, reposted or bookmarked, cached inreply_contexts.
Known gaps
From the v2.8.0 review, "Not done":
- The reply-context
h-citedoesn't carry the cached author h-card, name or published time. - Photo alt text falls back to "Image" when a photo-only post has no caption.
- Person tags (
p-category h-card) aren't supported. - Webmentions received before v2.8.0 keep NULL source details until they are re-validated.
WebSub
Spec: https://www.w3.org/TR/websub/. Decision: ADR-071, including its v2.10.1 amendments, which replace the original text where they differ.
Topics
All absolute URLs under SITE_URL:
| Topic | Content type | Updated when |
|---|---|---|
/ |
text/html | A public non-like note changes |
/feed.atom, /feed.rss, /feed.json |
feed types | A public non-like note changes |
/likes |
text/html | A public like changes |
/tag/<name> |
text/html | A note with that tag (before or after the change) changes |
/feed.xml (legacy) advertises /feed.rss as its rel=self. The
negotiated /feed advertises the topic of the format it served.
Discovery
Topic responses (200/304) carry Link: <hub>; rel="hub" and
Link: <topic>; rel="self". The same links are embedded: <link rel="hub">
and <link rel="self"> in HTML heads, a hub <link> in Atom, an
<atom:link rel="hub"> in RSS, and "hubs" in JSON Feed. All of this
happens only while WEBSUB_ENABLED (default true).
On change
websub.note_changed() runs on create, edit, publish/unpublish, delete and
undelete, and once photos are attached. If the note is or was public, it:
- bumps
content_version, which is in the feed cache key and the ETag, so every process serves fresh feeds at once; - queues one delivery per active subscription for each affected topic. Only the latest content is sent: a newer delivery replaces a waiting one.
Drafts that stay drafts change nothing.
Hub
POST /websub, form-encoded only (case-insensitive check):
-
hub.modeissubscribeorunsubscribe. -
hub.topicmust be one of the topics above, after decoding unreserved percent-encodings and comparing scheme and host case-insensitively. Tag topics must name an existing tag. -
hub.callbackmust:- be printable, at most 2,048 characters, and absolute http(s);
- not be on this site;
- use port 80, 443 or 1024 and up;
- pass the SSRF check.
It is stored with an IDNA host.
-
hub.secretmust be non-empty and under 200 bytes.hub.lease_secondsis clamped to 1 to 30 days (default 10). Both are ignored for unsubscriptions. -
The hub answers 202, then verifies later. Errors are 4xx with a plain-text reason.
Limits (429 when hit):
- 30 requests per minute per client IP (shared by all processes; needs
TRUSTED_PROXY=truebehind a proxy); - at most 200 pending verifications;
WEBSUB_MAX_SUBSCRIPTIONS(1000) in total;WEBSUB_MAX_PER_HOST(25) per callback IP address.
The limits count pending requests too. Renewals are always allowed.
WebSubWorker (file lock, poll WEBSUB_POLL_INTERVAL 5 s) handles each
queued item on its own, so one bad item can't stop it:
- Verification of intent (5.3):
GETthe callback (existing query kept) withhub.mode, the topic exactly as requested,hub.challengeandhub.lease_seconds. Success is a 2xx whose body equals the challenge. Replies over 4 KB and redirects fail. Network/DNS errors and 5xx are retried after 1, 5 and 15 minutes. A request id per intent (migration 017) stops an older verification from consuming a newer request. A subscription row exists only once verified. - Distribution (7): the topic is rendered inside the app, once per pass,
and POSTed with its
Content-Type,Linkrel=hub/self, andX-Hub-Signature: sha256=<HMAC>when there is a secret.- 2xx: success.
- 410: removes the subscription.
- Other failures: retried after 1 min, 5 min, 30 min, 2 h and 6 h, then that delivery is dropped. The subscription stays until its lease ends.
- A topic that renders 404/410 ends its subscriptions with a
hub.mode=deniednotice.
- Expired subscriptions are pruned hourly.
Outbound requests use safe_get / safe_post: pinned, no redirects,
10 s hard deadline.
External hub: with WEBSUB_HUB_URL set, StarPunk advertises that hub
and sends it a hub.mode=publish ping per changed topic. Its own /websub
refuses new subscriptions but still accepts unsubscriptions.
The admin dashboard shows subscribers per topic and deliveries being retried.
Known limits
- The hub serves only this site's topics, by design. websub.rocks' hub tests use their own topics, so they can't be run against it.
- Deliveries always carry the full topic content (no diffs). Fat pings aren't sent to external hubs.
Feeds
Decisions: ADR-014, ADR-038, ADR-059.
| URL | Format |
|---|---|
/feed.rss |
RSS 2.0 (with atom:link rel=self and Media RSS) |
/feed.xml |
Legacy alias of /feed.rss |
/feed.atom |
Atom 1.0 |
/feed.json |
JSON Feed 1.1 |
/feed |
Content negotiation on Accept (RSS default; 406 with X-Available-Formats if nothing fits; Vary: Accept) |
/opml.xml |
OPML 2.0 listing the three feeds |
- Feeds contain the newest
FEED_MAX_ITEMS(50) published notes, with full HTML content and media. Likes are excluded, as on the home page. Reposts and bookmarks start with a "🔁 Reposted" / "🔖 Bookmarked" line. - Feeds are cached for
FEED_CACHE_SECONDS(300). The cache key and ETag includecontent_version, so any change to a public note invalidates them in every process.If-None-Matchgets 304. base.htmladvertises the negotiated/feed(as RSS) and the OPML file withrel=alternate. Atom and JSON Feed are listed in the OPML but have no<link rel=alternate>of their own.- WebSub hub links are embedded in each feed (see above).
Gaps: there are no per-tag feeds (tag pages are h-feeds and WebSub topics,
but have no RSS/Atom/JSON equivalent). The /likes page has no feed.