Files
StarPunk/docs/design/v2.6.9/2026-11-03-webmention-processing-fixes.md
PhilandClaude Opus 5.5 106495d114 fix: Webmention processing works end to end (v2.6.9)
- Normalise mf2 author values: u-photo with alt text is a dict in mf2py and
  broke the database update for most real senders
- Pass str(response.url) to urljoin during endpoint discovery; the TypeError
  marked every target as having no endpoint. Retry on discovery errors
- Hold a cross-process flock so only one gunicorn process runs the worker
- Keep author URL/photo only if http(s) (XSS via javascript: URLs)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 07:53:19 -07:00

1.8 KiB

v2.6.9: Webmention processing fixes

Found after v2.6.8 let the worker run in production, and by a review of the implementation against the W3C Webmention Recommendation.

Fixes

  1. Author photo with alt text (webmention_processor.extract_author). mf2py returns u-photo with an alt attribute as {'value': url, 'alt': ...}; the dict was bound to SQLite and the update failed. _first_str() reduces mf2 values to their value.
  2. Endpoint discovery (WebmentionWorker._discover_endpoint). response.url is an httpx.URL; urljoin(str, URL) raised TypeError, which was caught and returned as "no endpoint", so every target was marked no_endpoint. Pass str(response.url). Fetch errors now propagate so the caller schedules a retry, instead of being treated as "no endpoint".
  3. Duplicate processing. gunicorn runs 4 processes, each with a worker, and queue items were claimed by SELECT then UPDATE. The worker now takes a non-blocking flock on <data>/webmention-worker.lock before processing; others wait and retry each poll, so one takes over if the holder dies. Orphan cleanup runs when the lock is acquired.
  4. XSS (spec 4.1). author_url and author_photo are kept only if they are absolute http(s) URLs; otherwise the source URL / no photo is used.

Tests

tests/test_webmention_worker.py: the send test uses a real httpx.Response (the old MagicMock with a string URL hid fix 2), plus tests for the lock, an alt-text photo and a javascript: author URL. All fail on v2.6.8.

Remaining spec gaps

Tracked separately: rel-token and document-order endpoint discovery, exact target match during verification, Accept header, re-sending on delete, 410 for deleted notes, SSRF re-validation on redirects, and the others listed in the compliance review.