docs: ADR-071 WebSub publishing with a built-in hub (proposed)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
PhilandClaude Opus 5.5 committed 2026-11-03 10:23:00 -07:00
1 parent c89c2cd5ca
commit 720a1e6d6f
1 file changed
+166
+166
View File
@@ -0,0 +1,166 @@
# ADR-071: WebSub publishing with a built-in hub
## Status
Proposed (v2.10.0, 2026-11-03)
## Context
Readers poll StarPunk's feeds. WebSub (W3C Recommendation,
https://www.w3.org/TR/websub/) lets readers that support it (Monocle,
Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and
have new content pushed to them within seconds.
WebSub has three roles. StarPunk is the **publisher**. Someone has to run
the **hub** that keeps subscriptions and delivers content. Public hubs
exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a
third-party dependency and some have shut down. The rest of this site is
self-hosted.
ADR-014 deferred WebSub for v1 as "complexity, external dependency".
Two existing problems matter here:
- Feeds are cached for `FEED_CACHE_SECONDS` (5 minutes) and nothing clears
the cache when a note changes. A hub fetching right after a change would
get stale content. Regular readers are also up to 5 minutes behind.
- The RSS feed is served at `/feed.rss` and `/feed.xml` (legacy alias), and
calls itself `/feed.xml`.
## Decision
StarPunk runs its own hub, **for its own topics only**, at `/websub`.
`WEBSUB_HUB_URL` can point at an external hub instead; then StarPunk
advertises that hub, pings it on changes, and its own `/websub` is off.
### 1. Topics
Absolute URLs under `SITE_URL`:
| Topic | Content type |
|---|---|
| `/` (h-feed) | text/html |
| `/feed.atom` | application/atom+xml |
| `/feed.rss` | application/rss+xml |
| `/feed.json` | application/feed+json |
| `/likes` | text/html |
| `/tag/<name>` | text/html |
`/feed.xml` and the negotiated `/feed` advertise the topic of the format
they serve as `rel=self` (`/feed.rss`, `/feed.atom`, `/feed.json`). The
RSS feed's own self link changes from `/feed.xml` to `/feed.rss`.
### 2. Discovery (publisher, spec 4)
Every topic response carries `Link: <hub>; rel="hub"` and exactly one
`Link: <topic>; rel="self"`. Embedded links as well: `<link rel="hub">` and
`<link rel="self">` in the HTML head of topic pages, `<link rel="hub">`
next to the existing self link in Atom, `<atom:link rel="hub">` in RSS, and
`"hubs": [{"type": "WebSub", "url": ...}]` in JSON Feed.
### 3. What changed, and when
`websub.note_changed(note, old_tags=())` is called wherever
`syndication.note_changed` is, plus on create. It:
1. clears the feed cache (all formats), and
2. if the note is or was public, queues delivery for `/`, the three
feeds, `/likes` if it is a like, and each `/tag/<name>` for its tags
before and after the change.
Drafts that stay drafts change nothing.
### 4. Hub endpoint (spec 5)
`POST /websub`, `application/x-www-form-urlencoded`:
- `hub.mode` subscribe or unsubscribe; `hub.topic` must be one of the
topics above (after normalising), else 400 with a plain-text reason;
`hub.callback` must be an absolute http(s) URL that passes the SSRF
check; `hub.secret` under 200 bytes; `hub.lease_seconds` clamped to
1 day to 30 days (default 10 days). Unknown parameters are ignored.
- Response: 202 Accepted, independent of what verification later finds.
- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions
per callback host and 1,000 in total; beyond that, 429.
### 5. Verification of intent (spec 5.3)
A background worker sends `GET <callback>` with the callback's query
string kept and `hub.mode`, `hub.topic`, `hub.challenge` (random, 32
bytes) and `hub.lease_seconds` appended. Success is a 2xx with the body
equal to the challenge; anything else, including redirects, fails and
leaves the subscription state unchanged. On success a subscribe becomes
active with `expires_at = now + lease`, and an unsubscribe deletes it.
A re-subscribe replaces the secret and lease only once verified.
### 6. Content distribution (spec 6, 7)
For each change, one delivery per active, unexpired subscription to the
topic. The worker renders the topic inside the app (no HTTP round trip
through Caddy), so subscribers get exactly what the public URL serves,
then `POST`s it to the callback with:
- `Content-Type` of the topic,
- `Link` headers with `rel=hub` and `rel=self`,
- `X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)>` if a secret was
given.
A 2xx is success. 410 Gone removes the subscription. Anything else is
retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a
subscription whose deliveries fail for 3 days is removed. Only the latest
content is sent: a queued delivery for a topic is replaced by a newer one.
### 7. Outbound safety
Callbacks are arbitrary URLs, so verification and delivery go through
`webmention_ssrf` (`safe_get`/`safe_post`): public addresses only, the
connection pinned to the checked address, no redirects, 10 s time limit,
1 MB response cap.
### 8. Data (migration 016)
- `websub_subscriptions(id, topic, callback, secret, lease_seconds,
state ['pending','active'], expires_at, created_at, verified_at,
last_success_at, last_error)`, unique on `(topic, callback)`.
- `websub_intents(id, topic, callback, mode, secret, lease_seconds,
attempts, next_attempt_at, created_at)`: verifications to do.
- `websub_deliveries(id, subscription_id, attempts, next_attempt_at,
last_error, created_at)`, unique on `subscription_id`.
### 9. Worker
The same pattern as the Webmention worker: one thread per gunicorn
process, a file lock so only one process works the queues, app context
around each poll. Expired subscriptions are pruned once a day.
### 10. Admin
The admin dashboard shows active subscribers per topic and failing
deliveries.
### 11. Configuration
`WEBSUB_ENABLED` (default true), `WEBSUB_HUB_URL` (default: built-in),
`WEBSUB_MAX_SUBSCRIPTIONS` (1000), `WEBSUB_MAX_PER_HOST` (25).
## Consequences
- Readers that support WebSub get posts and edits within seconds.
- Feeds are fresh right after a change for every reader, because the cache
is cleared.
- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF
protections, per-host and total limits, and rate limiting bound that.
- websub.rocks' hub tests use websub.rocks' own topics, which this hub
refuses by design. Testing uses unit tests of each spec section and a
real subscription from a reader after deploy.
- The RSS self link changes to `/feed.rss`. Readers keep whichever URL
they subscribed to; both keep working.
## Alternatives considered
- **Public hub only.** Least code, but a third-party dependency for a
self-hosted site. Still available through `WEBSUB_HUB_URL`.
- **A separate hub service** (another container). More to deploy and
keep updated, and few maintained hubs exist.
- **Accepting any topic** (a general-purpose hub). Not needed, and it
would let the hub be used to fetch and push arbitrary URLs.