From 720a1e6d6f0f4b11137ffc650b52a5a4645b90a9 Mon Sep 17 00:00:00 2001 From: Phil Date: Tue, 3 Nov 2026 10:23:00 -0700 Subject: [PATCH] docs: ADR-071 WebSub publishing with a built-in hub (proposed) Co-Authored-By: Claude Opus 5.5 --- docs/decisions/ADR-071-websub.md | 166 +++++++++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 docs/decisions/ADR-071-websub.md diff --git a/docs/decisions/ADR-071-websub.md b/docs/decisions/ADR-071-websub.md new file mode 100644 index 0000000..0e4743f --- /dev/null +++ b/docs/decisions/ADR-071-websub.md @@ -0,0 +1,166 @@ +# ADR-071: WebSub publishing with a built-in hub + +## Status + +Proposed (v2.10.0, 2026-11-03) + +## Context + +Readers poll StarPunk's feeds. WebSub (W3C Recommendation, +https://www.w3.org/TR/websub/) lets readers that support it (Monocle, +Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and +have new content pushed to them within seconds. + +WebSub has three roles. StarPunk is the **publisher**. Someone has to run +the **hub** that keeps subscriptions and delivers content. Public hubs +exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a +third-party dependency and some have shut down. The rest of this site is +self-hosted. + +ADR-014 deferred WebSub for v1 as "complexity, external dependency". + +Two existing problems matter here: + +- Feeds are cached for `FEED_CACHE_SECONDS` (5 minutes) and nothing clears + the cache when a note changes. A hub fetching right after a change would + get stale content. Regular readers are also up to 5 minutes behind. +- The RSS feed is served at `/feed.rss` and `/feed.xml` (legacy alias), and + calls itself `/feed.xml`. + +## Decision + +StarPunk runs its own hub, **for its own topics only**, at `/websub`. +`WEBSUB_HUB_URL` can point at an external hub instead; then StarPunk +advertises that hub, pings it on changes, and its own `/websub` is off. + +### 1. Topics + +Absolute URLs under `SITE_URL`: + +| Topic | Content type | +|---|---| +| `/` (h-feed) | text/html | +| `/feed.atom` | application/atom+xml | +| `/feed.rss` | application/rss+xml | +| `/feed.json` | application/feed+json | +| `/likes` | text/html | +| `/tag/` | text/html | + +`/feed.xml` and the negotiated `/feed` advertise the topic of the format +they serve as `rel=self` (`/feed.rss`, `/feed.atom`, `/feed.json`). The +RSS feed's own self link changes from `/feed.xml` to `/feed.rss`. + +### 2. Discovery (publisher, spec 4) + +Every topic response carries `Link: ; rel="hub"` and exactly one +`Link: ; rel="self"`. Embedded links as well: `` and +`` in the HTML head of topic pages, `` +next to the existing self link in Atom, `` in RSS, and +`"hubs": [{"type": "WebSub", "url": ...}]` in JSON Feed. + +### 3. What changed, and when + +`websub.note_changed(note, old_tags=())` is called wherever +`syndication.note_changed` is, plus on create. It: + +1. clears the feed cache (all formats), and +2. if the note is or was public, queues delivery for `/`, the three + feeds, `/likes` if it is a like, and each `/tag/` for its tags + before and after the change. + +Drafts that stay drafts change nothing. + +### 4. Hub endpoint (spec 5) + +`POST /websub`, `application/x-www-form-urlencoded`: + +- `hub.mode` subscribe or unsubscribe; `hub.topic` must be one of the + topics above (after normalising), else 400 with a plain-text reason; + `hub.callback` must be an absolute http(s) URL that passes the SSRF + check; `hub.secret` under 200 bytes; `hub.lease_seconds` clamped to + 1 day to 30 days (default 10 days). Unknown parameters are ignored. +- Response: 202 Accepted, independent of what verification later finds. +- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions + per callback host and 1,000 in total; beyond that, 429. + +### 5. Verification of intent (spec 5.3) + +A background worker sends `GET ` with the callback's query +string kept and `hub.mode`, `hub.topic`, `hub.challenge` (random, 32 +bytes) and `hub.lease_seconds` appended. Success is a 2xx with the body +equal to the challenge; anything else, including redirects, fails and +leaves the subscription state unchanged. On success a subscribe becomes +active with `expires_at = now + lease`, and an unsubscribe deletes it. +A re-subscribe replaces the secret and lease only once verified. + +### 6. Content distribution (spec 6, 7) + +For each change, one delivery per active, unexpired subscription to the +topic. The worker renders the topic inside the app (no HTTP round trip +through Caddy), so subscribers get exactly what the public URL serves, +then `POST`s it to the callback with: + +- `Content-Type` of the topic, +- `Link` headers with `rel=hub` and `rel=self`, +- `X-Hub-Signature: sha256=` if a secret was + given. + +A 2xx is success. 410 Gone removes the subscription. Anything else is +retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a +subscription whose deliveries fail for 3 days is removed. Only the latest +content is sent: a queued delivery for a topic is replaced by a newer one. + +### 7. Outbound safety + +Callbacks are arbitrary URLs, so verification and delivery go through +`webmention_ssrf` (`safe_get`/`safe_post`): public addresses only, the +connection pinned to the checked address, no redirects, 10 s time limit, +1 MB response cap. + +### 8. Data (migration 016) + +- `websub_subscriptions(id, topic, callback, secret, lease_seconds, + state ['pending','active'], expires_at, created_at, verified_at, + last_success_at, last_error)`, unique on `(topic, callback)`. +- `websub_intents(id, topic, callback, mode, secret, lease_seconds, + attempts, next_attempt_at, created_at)`: verifications to do. +- `websub_deliveries(id, subscription_id, attempts, next_attempt_at, + last_error, created_at)`, unique on `subscription_id`. + +### 9. Worker + +The same pattern as the Webmention worker: one thread per gunicorn +process, a file lock so only one process works the queues, app context +around each poll. Expired subscriptions are pruned once a day. + +### 10. Admin + +The admin dashboard shows active subscribers per topic and failing +deliveries. + +### 11. Configuration + +`WEBSUB_ENABLED` (default true), `WEBSUB_HUB_URL` (default: built-in), +`WEBSUB_MAX_SUBSCRIPTIONS` (1000), `WEBSUB_MAX_PER_HOST` (25). + +## Consequences + +- Readers that support WebSub get posts and edits within seconds. +- Feeds are fresh right after a change for every reader, because the cache + is cleared. +- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF + protections, per-host and total limits, and rate limiting bound that. +- websub.rocks' hub tests use websub.rocks' own topics, which this hub + refuses by design. Testing uses unit tests of each spec section and a + real subscription from a reader after deploy. +- The RSS self link changes to `/feed.rss`. Readers keep whichever URL + they subscribed to; both keep working. + +## Alternatives considered + +- **Public hub only.** Least code, but a third-party dependency for a + self-hosted site. Still available through `WEBSUB_HUB_URL`. +- **A separate hub service** (another container). More to deploy and + keep updated, and few maintained hubs exist. +- **Accepting any topic** (a general-purpose hub). Not needed, and it + would let the hub be used to fetch and push arbitrary URLs.