- Per-item error handling in the worker; hard time limits on requests - Notes with photos announced after attach; feed cache/ETag follow content_version - Keep failing subscriptions until lease end; retry DNS and render errors; hub.mode=denied for gone topics - Spec 5.1: decode unreserved chars, echo requested topic, ignore lease and secret on unsubscribe, case-insensitive Content-Type, refuse empty secret - Limits count pending requests per callback address; callback validation; request_id guards the verification race (migration 017) - Render once per topic per pass; likes only push /likes; Vary: Accept Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
8.9 KiB
ADR-071: WebSub publishing with a built-in hub
Status
Accepted (v2.10.0, 2026-11-03)
Context
Readers poll StarPunk's feeds. WebSub (W3C Recommendation, https://www.w3.org/TR/websub/) lets readers that support it (Monocle, Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and have new content pushed to them within seconds.
WebSub has three roles. StarPunk is the publisher. Someone has to run the hub that keeps subscriptions and delivers content. Public hubs exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a third-party dependency and some have shut down. The rest of this site is self-hosted.
ADR-014 deferred WebSub for v1 as "complexity, external dependency".
Two existing problems matter here:
- Feeds are cached for
FEED_CACHE_SECONDS(5 minutes) and nothing clears the cache when a note changes. A hub fetching right after a change would get stale content. Regular readers are also up to 5 minutes behind. - The RSS feed is served at
/feed.rssand/feed.xml(legacy alias), and calls itself/feed.xml.
Decision
StarPunk runs its own hub, for its own topics only, at /websub.
WEBSUB_HUB_URL can point at an external hub instead; then StarPunk
advertises that hub, pings it on changes, and its own /websub is off.
1. Topics
Absolute URLs under SITE_URL:
| Topic | Content type |
|---|---|
/ (h-feed) |
text/html |
/feed.atom |
application/atom+xml |
/feed.rss |
application/rss+xml |
/feed.json |
application/feed+json |
/likes |
text/html |
/tag/<name> |
text/html |
/feed.xml and the negotiated /feed advertise the topic of the format
they serve as rel=self (/feed.rss, /feed.atom, /feed.json). The
RSS feed's own self link changes from /feed.xml to /feed.rss.
2. Discovery (publisher, spec 4)
Every topic response carries Link: <hub>; rel="hub" and exactly one
Link: <topic>; rel="self". Embedded links as well: <link rel="hub"> and
<link rel="self"> in the HTML head of topic pages, <link rel="hub">
next to the existing self link in Atom, <atom:link rel="hub"> in RSS, and
"hubs": [{"type": "WebSub", "url": ...}] in JSON Feed.
3. What changed, and when
websub.note_changed(note, old_tags=()) is called wherever
syndication.note_changed is, plus on create. It:
- clears the feed cache (all formats), and
- if the note is or was public, queues delivery for
/, the three feeds,/likesif it is a like, and each/tag/<name>for its tags before and after the change.
Drafts that stay drafts change nothing.
4. Hub endpoint (spec 5)
POST /websub, application/x-www-form-urlencoded:
hub.modesubscribe or unsubscribe;hub.topicmust be one of the topics above (after normalising), else 400 with a plain-text reason;hub.callbackmust be an absolute http(s) URL that passes the SSRF check;hub.secretunder 200 bytes;hub.lease_secondsclamped to 1 day to 30 days (default 10 days). Unknown parameters are ignored.- Response: 202 Accepted, independent of what verification later finds.
- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions per callback host and 1,000 in total; beyond that, 429.
5. Verification of intent (spec 5.3)
A background worker sends GET <callback> with the callback's query
string kept and hub.mode, hub.topic, hub.challenge (random, 32
bytes) and hub.lease_seconds appended. Success is a 2xx with the body
equal to the challenge; anything else, including redirects, fails and
leaves the subscription state unchanged. On success a subscribe becomes
active with expires_at = now + lease, and an unsubscribe deletes it.
A re-subscribe replaces the secret and lease only once verified.
6. Content distribution (spec 6, 7)
For each change, one delivery per active, unexpired subscription to the
topic. The worker renders the topic inside the app (no HTTP round trip
through Caddy), so subscribers get exactly what the public URL serves,
then POSTs it to the callback with:
Content-Typeof the topic,Linkheaders withrel=hubandrel=self,X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)>if a secret was given.
A 2xx is success. 410 Gone removes the subscription. Anything else is retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a subscription whose deliveries fail for 3 days is removed. Only the latest content is sent: a queued delivery for a topic is replaced by a newer one.
7. Outbound safety
Callbacks are arbitrary URLs, so verification and delivery go through
webmention_ssrf (safe_get/safe_post): public addresses only, the
connection pinned to the checked address, no redirects, 10 s time limit,
1 MB response cap.
8. Data (migration 016)
websub_subscriptions(id, topic, callback, secret, lease_seconds, state ['pending','active'], expires_at, created_at, verified_at, last_success_at, last_error), unique on(topic, callback).websub_intents(id, topic, callback, mode, secret, lease_seconds, attempts, next_attempt_at, created_at): verifications to do.websub_deliveries(id, subscription_id, attempts, next_attempt_at, last_error, created_at), unique onsubscription_id.
9. Worker
The same pattern as the Webmention worker: one thread per gunicorn process, a file lock so only one process works the queues, app context around each poll. Expired subscriptions are pruned once a day.
10. Admin
The admin dashboard shows active subscribers per topic and failing deliveries.
11. Configuration
WEBSUB_ENABLED (default true), WEBSUB_HUB_URL (default: built-in),
WEBSUB_MAX_SUBSCRIPTIONS (1000), WEBSUB_MAX_PER_HOST (25).
Consequences
- Readers that support WebSub get posts and edits within seconds.
- Feeds are fresh right after a change for every reader, because the cache is cleared.
- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF protections, per-host and total limits, and rate limiting bound that.
- websub.rocks' hub tests use websub.rocks' own topics, which this hub refuses by design. Testing uses unit tests of each spec section and a real subscription from a reader after deploy.
- The RSS self link changes to
/feed.rss. Readers keep whichever URL they subscribed to; both keep working.
Alternatives considered
- Public hub only. Least code, but a third-party dependency for a
self-hosted site. Still available through
WEBSUB_HUB_URL. - A separate hub service (another container). More to deploy and keep updated, and few maintained hubs exist.
- Accepting any topic (a general-purpose hub). Not needed, and it would let the hub be used to fetch and push arbitrary URLs.
Amendments (v2.10.1, 2026-11-03)
An adversarial review of v2.10.0 against the Recommendation led to these changes; where they differ, they replace the text above.
- Worker robustness. Each queued item is handled on its own: an unexpected error drops (verification) or retries (delivery) that item only. Before, a callback answering with a large page or containing a control character stopped the worker for good.
- Time limits.
safe_getandsafe_postclose the connection when the whole exchange exceeds its time limit (10 s), not just a single read. Verification replies over 4 KB fail; delivery responses are read up to 4 KB. - Leases are kept (spec 7). A failing subscriber is not removed; its
deliveries are retried and then given up, and the next change is
delivered again. Only lease expiry, 410 Gone or unsubscribing end a
subscription. DNS failures are retried. A topic that renders 404/410
ends its subscriptions with a
hub.mode=deniednotice; other render errors are retried. - Photos. A note with photos is announced once they are attached
(
attach_media_to_notenotifies), and the feed cache key and ETag includecontent_version, so photo and caption changes reach feeds and subscribers. ETags no longer depend on the feed's build time. - Subscription requests (spec 5.1). Unreserved percent-encodings are
decoded; the Content-Type check is case-insensitive;
hub.secretandhub.lease_secondsare ignored for unsubscriptions; an empty secret is refused; verification echoeshub.topicexactly as requested (requested_topic); unsubscribing works while an external hub is configured. Callbacks must be printable, at most 2,048 characters, not on this site, on port 80, 443 or 1024+, and are stored with an IDNA host. - Limits count pending requests as well as subscriptions, per callback
address (so subdomains on one server share a limit), with at most 200
pending requests. Rate limiting needs
TRUSTED_PROXY=truebehind Caddy. - Races. Each request has a
request_id; verifying an older request doesn't consume a newer one (migration 017). - Less work. A topic is rendered once per worker pass; likes only
update
/likesand tag pages. - Corrections: there is no
statecolumn (a subscription row exists only once verified); expired leases are pruned hourly.