Files
StarPunk/docs/decisions/ADR-071-websub.md
T
PhilandClaude Opus 5.5 b520b8b4d3 feat: WebSub with a built-in hub (v2.10.0)
ADR-071. Topics (/, /feed.atom, /feed.rss, /feed.json, /likes, /tag/<name>)
advertise the hub and themselves in Link headers and embedded links. Note
changes bump a content version (fresh feeds in every process) and queue
deliveries. /websub accepts subscriptions for these topics only; the worker
verifies intent, delivers signed content with retries, and prunes expired
leases, all through the SSRF-safe pinned fetcher. WEBSUB_HUB_URL switches
to an external hub (publish pings). Also: the memory monitor no longer
stops after the first request; RSS self link is /feed.rss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:33:41 -07:00

6.6 KiB

ADR-071: WebSub publishing with a built-in hub

Status

Accepted (v2.10.0, 2026-11-03)

Context

Readers poll StarPunk's feeds. WebSub (W3C Recommendation, https://www.w3.org/TR/websub/) lets readers that support it (Monocle, Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and have new content pushed to them within seconds.

WebSub has three roles. StarPunk is the publisher. Someone has to run the hub that keeps subscriptions and delivers content. Public hubs exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a third-party dependency and some have shut down. The rest of this site is self-hosted.

ADR-014 deferred WebSub for v1 as "complexity, external dependency".

Two existing problems matter here:

  • Feeds are cached for FEED_CACHE_SECONDS (5 minutes) and nothing clears the cache when a note changes. A hub fetching right after a change would get stale content. Regular readers are also up to 5 minutes behind.
  • The RSS feed is served at /feed.rss and /feed.xml (legacy alias), and calls itself /feed.xml.

Decision

StarPunk runs its own hub, for its own topics only, at /websub. WEBSUB_HUB_URL can point at an external hub instead; then StarPunk advertises that hub, pings it on changes, and its own /websub is off.

1. Topics

Absolute URLs under SITE_URL:

Topic Content type
/ (h-feed) text/html
/feed.atom application/atom+xml
/feed.rss application/rss+xml
/feed.json application/feed+json
/likes text/html
/tag/<name> text/html

/feed.xml and the negotiated /feed advertise the topic of the format they serve as rel=self (/feed.rss, /feed.atom, /feed.json). The RSS feed's own self link changes from /feed.xml to /feed.rss.

2. Discovery (publisher, spec 4)

Every topic response carries Link: <hub>; rel="hub" and exactly one Link: <topic>; rel="self". Embedded links as well: <link rel="hub"> and <link rel="self"> in the HTML head of topic pages, <link rel="hub"> next to the existing self link in Atom, <atom:link rel="hub"> in RSS, and "hubs": [{"type": "WebSub", "url": ...}] in JSON Feed.

3. What changed, and when

websub.note_changed(note, old_tags=()) is called wherever syndication.note_changed is, plus on create. It:

  1. clears the feed cache (all formats), and
  2. if the note is or was public, queues delivery for /, the three feeds, /likes if it is a like, and each /tag/<name> for its tags before and after the change.

Drafts that stay drafts change nothing.

4. Hub endpoint (spec 5)

POST /websub, application/x-www-form-urlencoded:

  • hub.mode subscribe or unsubscribe; hub.topic must be one of the topics above (after normalising), else 400 with a plain-text reason; hub.callback must be an absolute http(s) URL that passes the SSRF check; hub.secret under 200 bytes; hub.lease_seconds clamped to 1 day to 30 days (default 10 days). Unknown parameters are ignored.
  • Response: 202 Accepted, independent of what verification later finds.
  • Limits: the Webmention rate limiter (per IP); at most 25 subscriptions per callback host and 1,000 in total; beyond that, 429.

5. Verification of intent (spec 5.3)

A background worker sends GET <callback> with the callback's query string kept and hub.mode, hub.topic, hub.challenge (random, 32 bytes) and hub.lease_seconds appended. Success is a 2xx with the body equal to the challenge; anything else, including redirects, fails and leaves the subscription state unchanged. On success a subscribe becomes active with expires_at = now + lease, and an unsubscribe deletes it. A re-subscribe replaces the secret and lease only once verified.

6. Content distribution (spec 6, 7)

For each change, one delivery per active, unexpired subscription to the topic. The worker renders the topic inside the app (no HTTP round trip through Caddy), so subscribers get exactly what the public URL serves, then POSTs it to the callback with:

  • Content-Type of the topic,
  • Link headers with rel=hub and rel=self,
  • X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)> if a secret was given.

A 2xx is success. 410 Gone removes the subscription. Anything else is retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a subscription whose deliveries fail for 3 days is removed. Only the latest content is sent: a queued delivery for a topic is replaced by a newer one.

7. Outbound safety

Callbacks are arbitrary URLs, so verification and delivery go through webmention_ssrf (safe_get/safe_post): public addresses only, the connection pinned to the checked address, no redirects, 10 s time limit, 1 MB response cap.

8. Data (migration 016)

  • websub_subscriptions(id, topic, callback, secret, lease_seconds, state ['pending','active'], expires_at, created_at, verified_at, last_success_at, last_error), unique on (topic, callback).
  • websub_intents(id, topic, callback, mode, secret, lease_seconds, attempts, next_attempt_at, created_at): verifications to do.
  • websub_deliveries(id, subscription_id, attempts, next_attempt_at, last_error, created_at), unique on subscription_id.

9. Worker

The same pattern as the Webmention worker: one thread per gunicorn process, a file lock so only one process works the queues, app context around each poll. Expired subscriptions are pruned once a day.

10. Admin

The admin dashboard shows active subscribers per topic and failing deliveries.

11. Configuration

WEBSUB_ENABLED (default true), WEBSUB_HUB_URL (default: built-in), WEBSUB_MAX_SUBSCRIPTIONS (1000), WEBSUB_MAX_PER_HOST (25).

Consequences

  • Readers that support WebSub get posts and edits within seconds.
  • Feeds are fresh right after a change for every reader, because the cache is cleared.
  • StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF protections, per-host and total limits, and rate limiting bound that.
  • websub.rocks' hub tests use websub.rocks' own topics, which this hub refuses by design. Testing uses unit tests of each spec section and a real subscription from a reader after deploy.
  • The RSS self link changes to /feed.rss. Readers keep whichever URL they subscribed to; both keep working.

Alternatives considered

  • Public hub only. Least code, but a third-party dependency for a self-hosted site. Still available through WEBSUB_HUB_URL.
  • A separate hub service (another container). More to deploy and keep updated, and few maintained hubs exist.
  • Accepting any topic (a general-purpose hub). Not needed, and it would let the hub be used to fetch and push arbitrary URLs.