Files
StarPunk/docs/decisions/ADR-071-websub.md
T
PhilandClaude Opus 5.5 b2043286fa fix: WebSub hardening from the adversarial review (v2.10.1)
- Per-item error handling in the worker; hard time limits on requests
- Notes with photos announced after attach; feed cache/ETag follow
  content_version
- Keep failing subscriptions until lease end; retry DNS and render errors;
  hub.mode=denied for gone topics
- Spec 5.1: decode unreserved chars, echo requested topic, ignore lease and
  secret on unsubscribe, case-insensitive Content-Type, refuse empty secret
- Limits count pending requests per callback address; callback validation;
  request_id guards the verification race (migration 017)
- Render once per topic per pass; likes only push /likes; Vary: Accept

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:23:36 -07:00

8.9 KiB

ADR-071: WebSub publishing with a built-in hub

Status

Accepted (v2.10.0, 2026-11-03)

Context

Readers poll StarPunk's feeds. WebSub (W3C Recommendation, https://www.w3.org/TR/websub/) lets readers that support it (Monocle, Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and have new content pushed to them within seconds.

WebSub has three roles. StarPunk is the publisher. Someone has to run the hub that keeps subscriptions and delivers content. Public hubs exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a third-party dependency and some have shut down. The rest of this site is self-hosted.

ADR-014 deferred WebSub for v1 as "complexity, external dependency".

Two existing problems matter here:

  • Feeds are cached for FEED_CACHE_SECONDS (5 minutes) and nothing clears the cache when a note changes. A hub fetching right after a change would get stale content. Regular readers are also up to 5 minutes behind.
  • The RSS feed is served at /feed.rss and /feed.xml (legacy alias), and calls itself /feed.xml.

Decision

StarPunk runs its own hub, for its own topics only, at /websub. WEBSUB_HUB_URL can point at an external hub instead; then StarPunk advertises that hub, pings it on changes, and its own /websub is off.

1. Topics

Absolute URLs under SITE_URL:

Topic Content type
/ (h-feed) text/html
/feed.atom application/atom+xml
/feed.rss application/rss+xml
/feed.json application/feed+json
/likes text/html
/tag/<name> text/html

/feed.xml and the negotiated /feed advertise the topic of the format they serve as rel=self (/feed.rss, /feed.atom, /feed.json). The RSS feed's own self link changes from /feed.xml to /feed.rss.

2. Discovery (publisher, spec 4)

Every topic response carries Link: <hub>; rel="hub" and exactly one Link: <topic>; rel="self". Embedded links as well: <link rel="hub"> and <link rel="self"> in the HTML head of topic pages, <link rel="hub"> next to the existing self link in Atom, <atom:link rel="hub"> in RSS, and "hubs": [{"type": "WebSub", "url": ...}] in JSON Feed.

3. What changed, and when

websub.note_changed(note, old_tags=()) is called wherever syndication.note_changed is, plus on create. It:

  1. clears the feed cache (all formats), and
  2. if the note is or was public, queues delivery for /, the three feeds, /likes if it is a like, and each /tag/<name> for its tags before and after the change.

Drafts that stay drafts change nothing.

4. Hub endpoint (spec 5)

POST /websub, application/x-www-form-urlencoded:

  • hub.mode subscribe or unsubscribe; hub.topic must be one of the topics above (after normalising), else 400 with a plain-text reason; hub.callback must be an absolute http(s) URL that passes the SSRF check; hub.secret under 200 bytes; hub.lease_seconds clamped to 1 day to 30 days (default 10 days). Unknown parameters are ignored.
  • Response: 202 Accepted, independent of what verification later finds.
  • Limits: the Webmention rate limiter (per IP); at most 25 subscriptions per callback host and 1,000 in total; beyond that, 429.

5. Verification of intent (spec 5.3)

A background worker sends GET <callback> with the callback's query string kept and hub.mode, hub.topic, hub.challenge (random, 32 bytes) and hub.lease_seconds appended. Success is a 2xx with the body equal to the challenge; anything else, including redirects, fails and leaves the subscription state unchanged. On success a subscribe becomes active with expires_at = now + lease, and an unsubscribe deletes it. A re-subscribe replaces the secret and lease only once verified.

6. Content distribution (spec 6, 7)

For each change, one delivery per active, unexpired subscription to the topic. The worker renders the topic inside the app (no HTTP round trip through Caddy), so subscribers get exactly what the public URL serves, then POSTs it to the callback with:

  • Content-Type of the topic,
  • Link headers with rel=hub and rel=self,
  • X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)> if a secret was given.

A 2xx is success. 410 Gone removes the subscription. Anything else is retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a subscription whose deliveries fail for 3 days is removed. Only the latest content is sent: a queued delivery for a topic is replaced by a newer one.

7. Outbound safety

Callbacks are arbitrary URLs, so verification and delivery go through webmention_ssrf (safe_get/safe_post): public addresses only, the connection pinned to the checked address, no redirects, 10 s time limit, 1 MB response cap.

8. Data (migration 016)

  • websub_subscriptions(id, topic, callback, secret, lease_seconds, state ['pending','active'], expires_at, created_at, verified_at, last_success_at, last_error), unique on (topic, callback).
  • websub_intents(id, topic, callback, mode, secret, lease_seconds, attempts, next_attempt_at, created_at): verifications to do.
  • websub_deliveries(id, subscription_id, attempts, next_attempt_at, last_error, created_at), unique on subscription_id.

9. Worker

The same pattern as the Webmention worker: one thread per gunicorn process, a file lock so only one process works the queues, app context around each poll. Expired subscriptions are pruned once a day.

10. Admin

The admin dashboard shows active subscribers per topic and failing deliveries.

11. Configuration

WEBSUB_ENABLED (default true), WEBSUB_HUB_URL (default: built-in), WEBSUB_MAX_SUBSCRIPTIONS (1000), WEBSUB_MAX_PER_HOST (25).

Consequences

  • Readers that support WebSub get posts and edits within seconds.
  • Feeds are fresh right after a change for every reader, because the cache is cleared.
  • StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF protections, per-host and total limits, and rate limiting bound that.
  • websub.rocks' hub tests use websub.rocks' own topics, which this hub refuses by design. Testing uses unit tests of each spec section and a real subscription from a reader after deploy.
  • The RSS self link changes to /feed.rss. Readers keep whichever URL they subscribed to; both keep working.

Alternatives considered

  • Public hub only. Least code, but a third-party dependency for a self-hosted site. Still available through WEBSUB_HUB_URL.
  • A separate hub service (another container). More to deploy and keep updated, and few maintained hubs exist.
  • Accepting any topic (a general-purpose hub). Not needed, and it would let the hub be used to fetch and push arbitrary URLs.

Amendments (v2.10.1, 2026-11-03)

An adversarial review of v2.10.0 against the Recommendation led to these changes; where they differ, they replace the text above.

  • Worker robustness. Each queued item is handled on its own: an unexpected error drops (verification) or retries (delivery) that item only. Before, a callback answering with a large page or containing a control character stopped the worker for good.
  • Time limits. safe_get and safe_post close the connection when the whole exchange exceeds its time limit (10 s), not just a single read. Verification replies over 4 KB fail; delivery responses are read up to 4 KB.
  • Leases are kept (spec 7). A failing subscriber is not removed; its deliveries are retried and then given up, and the next change is delivered again. Only lease expiry, 410 Gone or unsubscribing end a subscription. DNS failures are retried. A topic that renders 404/410 ends its subscriptions with a hub.mode=denied notice; other render errors are retried.
  • Photos. A note with photos is announced once they are attached (attach_media_to_note notifies), and the feed cache key and ETag include content_version, so photo and caption changes reach feeds and subscribers. ETags no longer depend on the feed's build time.
  • Subscription requests (spec 5.1). Unreserved percent-encodings are decoded; the Content-Type check is case-insensitive; hub.secret and hub.lease_seconds are ignored for unsubscriptions; an empty secret is refused; verification echoes hub.topic exactly as requested (requested_topic); unsubscribing works while an external hub is configured. Callbacks must be printable, at most 2,048 characters, not on this site, on port 80, 443 or 1024+, and are stored with an IDNA host.
  • Limits count pending requests as well as subscriptions, per callback address (so subdomains on one server share a limit), with at most 200 pending requests. Rate limiting needs TRUSTED_PROXY=true behind Caddy.
  • Races. Each request has a request_id; verifying an older request doesn't consume a newer one (migration 017).
  • Less work. A topic is rendered once per worker pass; likes only update /likes and tag pages.
  • Corrections: there is no state column (a subscription row exists only once verified); expired leases are pruned hourly.