ADR-071. Topics (/, /feed.atom, /feed.rss, /feed.json, /likes, /tag/<name>) advertise the hub and themselves in Link headers and embedded links. Note changes bump a content version (fresh feeds in every process) and queue deliveries. /websub accepts subscriptions for these topics only; the worker verifies intent, delivers signed content with retries, and prunes expired leases, all through the SSRF-safe pinned fetcher. WEBSUB_HUB_URL switches to an external hub (publish pings). Also: the memory monitor no longer stops after the first request; RSS self link is /feed.rss. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
6.6 KiB
ADR-071: WebSub publishing with a built-in hub
Status
Accepted (v2.10.0, 2026-11-03)
Context
Readers poll StarPunk's feeds. WebSub (W3C Recommendation, https://www.w3.org/TR/websub/) lets readers that support it (Monocle, Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and have new content pushed to them within seconds.
WebSub has three roles. StarPunk is the publisher. Someone has to run the hub that keeps subscriptions and delivers content. Public hubs exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a third-party dependency and some have shut down. The rest of this site is self-hosted.
ADR-014 deferred WebSub for v1 as "complexity, external dependency".
Two existing problems matter here:
- Feeds are cached for
FEED_CACHE_SECONDS(5 minutes) and nothing clears the cache when a note changes. A hub fetching right after a change would get stale content. Regular readers are also up to 5 minutes behind. - The RSS feed is served at
/feed.rssand/feed.xml(legacy alias), and calls itself/feed.xml.
Decision
StarPunk runs its own hub, for its own topics only, at /websub.
WEBSUB_HUB_URL can point at an external hub instead; then StarPunk
advertises that hub, pings it on changes, and its own /websub is off.
1. Topics
Absolute URLs under SITE_URL:
| Topic | Content type |
|---|---|
/ (h-feed) |
text/html |
/feed.atom |
application/atom+xml |
/feed.rss |
application/rss+xml |
/feed.json |
application/feed+json |
/likes |
text/html |
/tag/<name> |
text/html |
/feed.xml and the negotiated /feed advertise the topic of the format
they serve as rel=self (/feed.rss, /feed.atom, /feed.json). The
RSS feed's own self link changes from /feed.xml to /feed.rss.
2. Discovery (publisher, spec 4)
Every topic response carries Link: <hub>; rel="hub" and exactly one
Link: <topic>; rel="self". Embedded links as well: <link rel="hub"> and
<link rel="self"> in the HTML head of topic pages, <link rel="hub">
next to the existing self link in Atom, <atom:link rel="hub"> in RSS, and
"hubs": [{"type": "WebSub", "url": ...}] in JSON Feed.
3. What changed, and when
websub.note_changed(note, old_tags=()) is called wherever
syndication.note_changed is, plus on create. It:
- clears the feed cache (all formats), and
- if the note is or was public, queues delivery for
/, the three feeds,/likesif it is a like, and each/tag/<name>for its tags before and after the change.
Drafts that stay drafts change nothing.
4. Hub endpoint (spec 5)
POST /websub, application/x-www-form-urlencoded:
hub.modesubscribe or unsubscribe;hub.topicmust be one of the topics above (after normalising), else 400 with a plain-text reason;hub.callbackmust be an absolute http(s) URL that passes the SSRF check;hub.secretunder 200 bytes;hub.lease_secondsclamped to 1 day to 30 days (default 10 days). Unknown parameters are ignored.- Response: 202 Accepted, independent of what verification later finds.
- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions per callback host and 1,000 in total; beyond that, 429.
5. Verification of intent (spec 5.3)
A background worker sends GET <callback> with the callback's query
string kept and hub.mode, hub.topic, hub.challenge (random, 32
bytes) and hub.lease_seconds appended. Success is a 2xx with the body
equal to the challenge; anything else, including redirects, fails and
leaves the subscription state unchanged. On success a subscribe becomes
active with expires_at = now + lease, and an unsubscribe deletes it.
A re-subscribe replaces the secret and lease only once verified.
6. Content distribution (spec 6, 7)
For each change, one delivery per active, unexpired subscription to the
topic. The worker renders the topic inside the app (no HTTP round trip
through Caddy), so subscribers get exactly what the public URL serves,
then POSTs it to the callback with:
Content-Typeof the topic,Linkheaders withrel=hubandrel=self,X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)>if a secret was given.
A 2xx is success. 410 Gone removes the subscription. Anything else is retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a subscription whose deliveries fail for 3 days is removed. Only the latest content is sent: a queued delivery for a topic is replaced by a newer one.
7. Outbound safety
Callbacks are arbitrary URLs, so verification and delivery go through
webmention_ssrf (safe_get/safe_post): public addresses only, the
connection pinned to the checked address, no redirects, 10 s time limit,
1 MB response cap.
8. Data (migration 016)
websub_subscriptions(id, topic, callback, secret, lease_seconds, state ['pending','active'], expires_at, created_at, verified_at, last_success_at, last_error), unique on(topic, callback).websub_intents(id, topic, callback, mode, secret, lease_seconds, attempts, next_attempt_at, created_at): verifications to do.websub_deliveries(id, subscription_id, attempts, next_attempt_at, last_error, created_at), unique onsubscription_id.
9. Worker
The same pattern as the Webmention worker: one thread per gunicorn process, a file lock so only one process works the queues, app context around each poll. Expired subscriptions are pruned once a day.
10. Admin
The admin dashboard shows active subscribers per topic and failing deliveries.
11. Configuration
WEBSUB_ENABLED (default true), WEBSUB_HUB_URL (default: built-in),
WEBSUB_MAX_SUBSCRIPTIONS (1000), WEBSUB_MAX_PER_HOST (25).
Consequences
- Readers that support WebSub get posts and edits within seconds.
- Feeds are fresh right after a change for every reader, because the cache is cleared.
- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF protections, per-host and total limits, and rate limiting bound that.
- websub.rocks' hub tests use websub.rocks' own topics, which this hub refuses by design. Testing uses unit tests of each spec section and a real subscription from a reader after deploy.
- The RSS self link changes to
/feed.rss. Readers keep whichever URL they subscribed to; both keep working.
Alternatives considered
- Public hub only. Least code, but a third-party dependency for a
self-hosted site. Still available through
WEBSUB_HUB_URL. - A separate hub service (another container). More to deploy and keep updated, and few maintained hubs exist.
- Accepting any topic (a general-purpose hub). Not needed, and it would let the hub be used to fetch and push arbitrary URLs.