Files
StarPunk/docs/decisions/ADR-063-webmention-implementation.md
PhilandClaude Opus 5.5 0716984958 docs: Bring all documentation up to date for v2.10.1
- architecture/: overview, technology stack, auth (OIDC + built-in
  IndieAuth), syndication (POSSE) rewritten; new database.md,
  deployment.md and indieweb-protocols.md; legacy stack marked historical
- decisions/: reconstructed ADR-060 and ADR-063; 31 status corrections
  (superseded auth ADRs, proposed -> accepted); INDEX rebuilt
- design/: reconstructed the v1.6.0 documents the webmention code cites;
  reports for 18 v2 releases without a design folder; INDEX rebuilt
- projectplan/: BACKLOG rewritten (shipped items removed, open issues from
  the reviews added); RELEASE.md for v2.0.0-v2.10.1; INDEX
- standards/: development setup (uv, tests, DEV_MODE), testing (CI test
  stage), git and release flow, versioning, coding standards, cookies
- README, CLAUDE.md, .env.example (every setting the code reads, with
  defaults) brought up to date
- 137 broken links repaired (old absolute paths, moved files); stale ADR
  references in code comments corrected (comments only)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:40:19 -07:00

14 KiB

ADR-063: Webmention Implementation

Status

Accepted (v1.6.0; revised v2.6.8, v2.6.9, v2.7.0, v2.7.2, v2.8.0, v2.10.1)

Reconstructed on 2026-11-03 from code, changelog and related documents; the original was lost. The v1.6.0 release was never committed: its code was recovered from the production image (v1.6.0-rc.2), but its tests, this ADR and the docs/design/v1.6.0/ folder (background-worker.md, api-contract.md, display-moderation.md, rate-limiting.md) were not in the image. This document describes the design as it exists in v2.10.1. The numbering of the decisions below is new, except that Decision 9 (SSRF protection) keeps the number the code cites.

Context

Webmention (W3C Recommendation, https://www.w3.org/TR/webmention/) is how IndieWeb sites tell each other that one page links to another. It is what makes replies, likes and reposts from other sites (and from Mastodon or Bluesky via Bridgy) show up on a note, and what tells the sites a note replies to or links to that it exists.

Both directions involve fetching URLs chosen by someone else, which is slow, can fail, and can be abused:

  • A receiver must fetch the source page to check that it really links to the target. Doing that inside the request lets anyone make StarPunk fetch arbitrary URLs synchronously, and ties up a server process.
  • A sender must discover each target's endpoint and POST to it, with retries when the other site is down.
  • Any fetch can be pointed (directly, by redirect or by DNS) at private addresses on the server's own network (SSRF).
  • Content from other sites is shown on our pages (XSS).

StarPunk is a single-user site on SQLite, run under gunicorn with several processes, with no job queue or message broker.

Decision

1. Endpoint and discovery

  • POST /webmention receives webmentions (form-encoded as the spec requires; JSON is accepted as an extension); GET /webmention returns a short description of the endpoint.
  • HTML responses advertise it with a Link: </webmention>; rel="webmention" header (added alongside other Link headers, not replacing them) and a <link rel="webmention"> in base.html.
  • WEBMENTION_ENABLED (default true) turns the feature off.

2. Receiving is asynchronous

The endpoint does only cheap, local checks and then answers 202 Accepted; verification happens later in the background worker.

Synchronous checks, each answered with 400 on failure: source and target are well-formed http(s) URLs of at most 2048 characters, they differ, and the target is a published note on this site (host compared case-insensitively and without default ports; #fragment dropped). Since v2.7.0 mentions of the home page or other pages are refused instead of being stored and never shown.

The mention is upserted into webmentions keyed by (source_url, target_url) with validation_status = 'pending', so a re-sent mention is re-verified rather than duplicated.

3. Background worker

One daemon thread (WebmentionWorker) handles both incoming verification and outgoing sending, using two SQLite tables as queues (webmentions, webmention_queue). No external queue is introduced.

  • Started by create_app() when WEBMENTION_WORKER_ENABLED is true (not in tests). It polls every WEBMENTION_POLL_INTERVAL seconds (10), handles up to WEBMENTION_BATCH_SIZE items (5) per pass, and backs off WEBMENTION_ERROR_BACKOFF seconds (60) after an unexpected error.
  • Each pass runs inside app.app_context() (v2.6.8: without it every item failed with "Working outside of application context"). The thread lives for the life of the process; it is not stopped on app-context teardown (v2.6.8: that stopped it after the first request).
  • Only one process works the queues (v2.6.9). Every gunicorn process starts a worker, but a worker only processes after taking a non-blocking flock on <data>/webmention-worker.lock. Others retry each poll, so one takes over if the holder dies.
  • On acquiring the lock it resets outgoing items stuck in sending for more than 5 minutes (orphans from a crash).
  • /health?detailed=true (authenticated) reports the worker's statistics.

4. Verifying a received webmention

Per Recommendation 3.2.2 and 3.2.4:

  • The source is fetched with an Accept header, following redirects (Decision 9), and decoded in its declared character set.
  • The source must link to the exact target URL, by media type: for HTML any link attribute (resolved against the final URL after redirects); for JSON any string value; for text/* a substring; other types never match. Fragments are ignored (v2.7.0).
  • The source is parsed with mf2py. The responding entry and its author are found with the authorship algorithm (v2.8.0); the mention type is one of reply, like, repost, bookmark or mention. The source's own u-url and dt-published are stored (source_post_url, source_published, migration 014) so the response links to and is dated by the original post.
  • If the source is gone (410, other 4xx) or no longer links, the mention is removed from display. If it can't be reached right now (network error, 5xx, temporary DNS failure), a previously valid mention stays up (v2.7.0, v2.10.1).
  • A webmention from one of the note's own crossposts (e.g. its GoToSocial copy returned through Bridgy) is ignored (v2.8.0).

5. Moderation and display

  • Valid mentions are approved automatically when WEBMENTION_AUTO_APPROVE is true (the default), otherwise they wait in pending. The admin can approve, hide or delete them at /admin/webmentions (single and bulk actions).
  • A re-sent mention keeps its moderation: hidden stays hidden, approved stays approved (v2.7.0).
  • Notes show only mentions that are both valid and approved, grouped by type. Replies are marked up as u-comment h-cite; likes, reposts, bookmarks and mentions as u-like, u-repost, u-bookmark, u-mention h-cites, each with the author's h-card (v2.8.0).

6. Sending

  • After a note is created, edited, deleted or restored, links are extracted from its rendered HTML: <a href> only, excluding links inside <pre> and <code>, the in_reply_to target first, then document order, at most 10 current links per operation. Each target is queued in webmention_queue (unique per source and target).
  • On edit, every target sent to before is notified again, including links the edit removed and an old reply target; on delete and undelete too (Recommendation 3.1.4, 3.1.5; v2.7.0). Re-sends are not capped.
  • When the responses a note displays change (a mention arrives, changes or is removed, or the admin approves, hides or deletes one that was shown), the note's previous webmentions are re-queued, as 3.1.4 recommends. An unchanged re-verification sends nothing, so two sites that both do this don't ping-pong (v2.7.2).
  • Endpoint discovery follows 3.1.2: the first Link header with a webmention rel token (RFC 8288 parsing, several headers, any case), otherwise the first <link> or <a> in document order; relative URLs resolve against the final URL. The endpoint is discovered afresh every time; there is no cache (v2.7.0).
  • The webmention is a form-encoded POST. Any 2xx is success; redirects are not followed (a redirected POST would become a GET). A target without an endpoint is marked no_endpoint.
  • Failures (network errors, 5xx, 429) are retried after 1 h, 6 h and 24 h; after the fourth failed attempt the item is failed.
  • The User-Agent is StarPunk/<version> Webmention (+site).

7. Deleted notes

A deleted note answers 410 Gone with a "This post has been deleted" page (soft-deleted notes, and hard-deleted notes that sent webmentions), so receivers can remove the mention (3.1.5; v2.7.0).

8. Content safety (XSS)

  • Snippets and reply content are sanitized with nh3, cut at 500 characters and sanitized again so no tag is left open (v2.7.0); the admin "Mentions" list shows them with tags stripped.
  • Author URLs and photos are kept only if they are absolute http(s) URLs; otherwise the source URL is used as the author link and no photo is shown (v2.6.9). mf2 values with alt are reduced to their URL.

9. SSRF protection

All outbound requests for webmentions (source fetches, endpoint discovery and the POST) go through starpunk/webmention_ssrf.py (Recommendation 4.2, 4.3, 4.5):

  • Only http(s). Every address a hostname resolves to (IPv4 and IPv6) must be public: private, loopback, link-local, multicast, reserved, unspecified, carrier-grade NAT (100.64.0.0/10), 0.0.0.0/8 and IPv4-mapped forms are blocked.
  • Redirects (at most 10) are followed by hand and each hop is checked (v2.7.0).
  • DNS pinning (v2.7.2): each request connects to the exact address that passed the check, with the original host in the Host header and as TLS SNI, so certificates are still verified against the hostname. A rebinding DNS server can't answer differently the second time. IPv4 is preferred when a host has both.
  • Responses are limited to 1 MB, and each fetch has a total time limit of 10 s that a slow, byte-by-byte response can't extend (v2.10.1).
  • DEV_MODE allows localhost for local testing.

The same guard is reused for IndieAuth client metadata fetches (ADR-064), Micropub remote photos (v2.9.0) and WebSub callbacks and deliveries (ADR-071).

10. Rate limiting

POST /webmention is rate-limited before any other work: by client IP (WEBMENTION_RATE_LIMIT_IP, 60) and by source domain (WEBMENTION_RATE_LIMIT_DOMAIN, 10) per WEBMENTION_RATE_LIMIT_WINDOW (60 s), answering 429 with Retry-After. Counters are an in-memory sliding-window log (starpunk/rate_limiter.py), so they are per server process. Behind a reverse proxy, TRUSTED_PROXY=true makes the first X-Forwarded-For address the client IP. WEBMENTION_RATE_LIMIT_ENABLED turns it off.

starpunk/rate_limiter.py and routes/webmention.py cite this as "ADR-064: Webmention Rate Limiting", a v1.6.0 ADR that was lost with this one. The number ADR-064 was later reused for OIDC login (ADR-064-oidc-admin-login-and-builtin-indieauth-server.md); the rate-limit decision is recorded here instead.

11. Data (migrations 010 and 014)

  • notes.in_reply_to
  • webmentions: source, target, target note, type, author name/URL/photo, snippet, validation_status (pending, valid, invalid, error), moderation_status (pending, approved, hidden, deleted), timestamps, and (migration 014) source_post_url, source_published
  • webmention_queue: outgoing items with status (pending, sending, sent, failed, no_endpoint), retry count and next retry time
  • reply_contexts: cached title, snippet and author of pages notes reply to

Consequences

Positive

  • Receiving is cheap and can't be used to make StarPunk fetch URLs while the sender waits.
  • No new infrastructure: the queues are SQLite tables and the worker is a thread, as for crossposting and WebSub.
  • Outbound fetching is hardened once and reused by every other feature that fetches user-supplied URLs.
  • Follows every MUST and SHOULD of the Recommendation after v2.7.2.

Negative

  • Mentions appear only after the next worker pass (seconds), not immediately.
  • One worker per site: throughput is limited, which is fine for a personal site.
  • Rate limits are per process and reset on restart.
  • error rows (unexpected failures) are not retried automatically.
  • Endpoint discovery ignores HTTP cache headers (spec 5.3); targets are only fetched when a note changes.
  • The 410 page is a generic tombstone without microformats.

Revisions

  • v2.6.8 — Worker runs inside an app context and is no longer stopped on teardown; before this, no webmention was ever processed (docs/design/v2.6.8/2026-11-03-webmention-worker-app-context.md).
  • v2.6.9 — Author photos with alt text; endpoint discovery crash fixed and discovery errors retried; flock so only one process works the queues; http(s)-only author links (docs/design/v2.6.9/2026-11-03-webmention-processing-fixes.md).
  • v2.7.0 — Full review against the Recommendation: published notes only, exact target match, RFC 8288 endpoint discovery, no discovery cache, re-sends on edit/delete/undelete, 410 for deleted notes, per-hop SSRF checks, size and time limits, moderation kept on re-send (docs/design/v2.7.0/2026-11-03-webmention-spec-compliance.md).
  • v2.7.2 — DNS pinning; re-send when displayed responses change (docs/design/v2.7.2/2026-11-03-webmention-shoulds.md).
  • v2.8.0 — Authorship algorithm, source URL and published time (migration 014), response microformats, own crossposts ignored (docs/design/v2.8.0/2026-11-03-microformats2-review.md).
  • v2.10.1 — Hard total time limit on fetches; temporary DNS failures treated as transient.

Alternatives Considered

  • Synchronous verification in the request: simpler, but slow, ties up a gunicorn process and turns the endpoint into a fetch-anything proxy. The Recommendation allows 202 for exactly this.
  • External service (webmention.io): no code to run, but mentions would live outside the site's database and moderation, against the self-hosted approach.
  • A job queue (Celery/RQ with Redis): new infrastructure for a few jobs a day.
  • Resolving and validating the host, then letting the HTTP client connect normally: open to DNS rebinding; replaced by pinning in v2.7.2.

References

  • W3C Webmention: https://www.w3.org/TR/webmention/
  • Code: starpunk/routes/webmention.py, starpunk/webmention_worker.py, starpunk/webmention_processor.py, starpunk/webmention_sender.py, starpunk/webmention_ssrf.py, starpunk/webmention_queries.py, starpunk/rate_limiter.py, migrations/010_add_webmentions.sql, migrations/014_webmention_source_details.sql
  • Tests: tests/test_webmention_spec.py, tests/test_webmention_worker.py
  • IndieWeb protocols
  • ADR-064 (OIDC login; reuses the SSRF guard), ADR-071 (WebSub; same guard and worker pattern)