Compare commits

..
5 Commits
Author SHA1 Message Date
Phil fa551de197 Merge branch 'fix/v2.10.1-websub-hardening' for v2.10.1 release
Build Container / build (push) Successful in 7m53s
2026-11-03 11:23:36 -07:00
PhilandClaude Opus 5.5 b2043286fa fix: WebSub hardening from the adversarial review (v2.10.1)
- Per-item error handling in the worker; hard time limits on requests
- Notes with photos announced after attach; feed cache/ETag follow
  content_version
- Keep failing subscriptions until lease end; retry DNS and render errors;
  hub.mode=denied for gone topics
- Spec 5.1: decode unreserved chars, echo requested topic, ignore lease and
  secret on unsubscribe, case-insensitive Content-Type, refuse empty secret
- Limits count pending requests per callback address; callback validation;
  request_id guards the verification race (migration 017)
- Render once per topic per pass; likes only push /likes; Vary: Accept

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 11:23:36 -07:00
Phil 409b8b31b4 Merge branch 'feat/v2.10.0-websub' for v2.10.0 release
Build Container / build (push) Successful in 7m29s
2026-11-03 10:33:41 -07:00
PhilandClaude Opus 5.5 b520b8b4d3 feat: WebSub with a built-in hub (v2.10.0)
ADR-071. Topics (/, /feed.atom, /feed.rss, /feed.json, /likes, /tag/<name>)
advertise the hub and themselves in Link headers and embedded links. Note
changes bump a content version (fresh feeds in every process) and queue
deliveries. /websub accepts subscriptions for these topics only; the worker
verifies intent, delivers signed content with retries, and prunes expired
leases, all through the SSRF-safe pinned fetcher. WEBSUB_HUB_URL switches
to an external hub (publish pings). Also: the memory monitor no longer
stops after the first request; RSS self link is /feed.rss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:33:41 -07:00
PhilandClaude Opus 5.5 720a1e6d6f docs: ADR-071 WebSub publishing with a built-in hub (proposed)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 10:23:00 -07:00
24 changed files with 2050 additions and 36 deletions

No files matched your search

+18
View File
@@ -125,6 +125,24 @@ WORKER_TIMEOUT=30
# Max requests per worker before restart (prevents memory leaks)
MAX_REQUESTS=1000
# Behind a reverse proxy (Caddy, nginx): take the client address from
# X-Forwarded-For. Without it every visitor shares the proxy's address, so one
# busy client can rate-limit everyone on /webmention and /websub.
# Only enable when the proxy sets X-Forwarded-For itself (Caddy does).
TRUSTED_PROXY=true
# =============================================================================
# WEBSUB (v2.10.0)
# =============================================================================
# Built-in hub at /websub for this site's feeds. Set WEBSUB_HUB_URL to use
# another hub instead (StarPunk then pings it on changes).
WEBSUB_ENABLED=true
# WEBSUB_HUB_URL=https://hub.example.com/
# WEBSUB_MAX_SUBSCRIPTIONS=1000
# Per callback server (resolved address)
# WEBSUB_MAX_PER_HOST=25
# =============================================================================
# DEVELOPMENT OPTIONS
# =============================================================================
+43
View File
@@ -7,6 +7,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
## [2.10.1] - 2026-11-03
WebSub fixes from an adversarial review of v2.10.0 against the [W3C Recommendation](https://www.w3.org/TR/websub/) (details in ADR-071, Amendments).
### Security
- One subscription request could stop the hub for good: a callback that answered verification with a large page, or a callback URL with a control character, crashed the worker on every pass. Each queued item is now handled on its own
- A subscriber sending its response one byte at a time could hold the hub for hours. Every request the hub (and webmention fetching) makes now has a hard time limit
- The subscription limits only counted verified subscriptions and could be dodged with subdomains. They now count pending requests too, per callback server address, and at most 200 requests can wait for verification
- Callbacks must be printable URLs of reasonable length, not on this site, and on ports 80, 443 or 1024 and up
### Fixed
- New posts with photos were sent to subscribers (and cached in the feeds) without their photos, and photo-only edits sent the old feed. Notes with photos are announced once the photos are attached, and the feed cache follows photo and caption changes
- A subscriber whose deliveries failed for three days was removed; WebSub says to keep it until its lease ends. Deliveries are retried and given up, the subscription stays
- A temporary DNS failure or an error rendering the feed lost the delivery instead of retrying it. A feed that no longer exists now ends its subscriptions with a `hub.mode=denied` notice
- Verification sent our normalised topic (`/tag/python`) instead of the one requested (`/tag/Python`), so strict subscribers refused it. Percent-encoded topics (`/%66eed.atom`) are now understood, the Content-Type check ignores case, unsubscribe requests ignore `hub.lease_seconds`, and an empty `hub.secret` is refused instead of turning signatures off
- A newer subscription request arriving while an older one was being verified was lost (migration 017)
- Unsubscribing works while `WEBSUB_HUB_URL` points to another hub
- Feed ETags differed between server processes, so conditional requests rarely matched; `/feed` now sends `Vary: Accept`
- A like no longer causes pushes of the home page and feeds, which don't show likes; each feed is rendered once per delivery round, not once per subscriber
- Internationalised callback domains can be verified
### Changed
- `.env.example` documents `TRUSTED_PROXY` (needed behind a reverse proxy for per-client rate limits) and the WebSub settings
## [2.10.0] - 2026-11-03
### Added
- **WebSub** ([ADR-071](docs/decisions/ADR-071-websub.md)): readers that support it get new posts, edits and deletions within seconds instead of polling. StarPunk runs its own hub at `/websub` for its own feeds: the home page, `/feed.atom`, `/feed.rss`, `/feed.json`, `/likes` and each tag page. Each advertises the hub and itself (`Link` headers, and links inside the page or feed). Subscriptions are verified with a challenge, leases expire, deliveries carry the full feed, signed with `X-Hub-Signature` when the subscriber gave a secret, and failures are retried for about nine hours. Requests to subscribers go through the same private-address protections as webmentions, and there are limits per subscriber host and in total. Set `WEBSUB_HUB_URL` to use another hub instead; `WEBSUB_ENABLED=false` turns it off
- The admin dashboard shows WebSub subscribers per feed
### Fixed
- Feeds could be up to five minutes out of date after posting, editing or deleting, and each server process cached its own copy. Any change to a public note now refreshes the feeds everywhere at once
- The memory monitor stopped after the first request (the same bug the webmention worker had before v2.6.8)
### Changed
- The RSS feed calls itself `/feed.rss` instead of the older `/feed.xml`. Both addresses still work
## [2.9.0] - 2026-11-03
Micropub review against the [W3C Recommendation](https://www.w3.org/TR/micropub/).
+206
View File
@@ -0,0 +1,206 @@
# ADR-071: WebSub publishing with a built-in hub
## Status
Accepted (v2.10.0, 2026-11-03)
## Context
Readers poll StarPunk's feeds. WebSub (W3C Recommendation,
https://www.w3.org/TR/websub/) lets readers that support it (Monocle,
Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and
have new content pushed to them within seconds.
WebSub has three roles. StarPunk is the **publisher**. Someone has to run
the **hub** that keeps subscriptions and delivers content. Public hubs
exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a
third-party dependency and some have shut down. The rest of this site is
self-hosted.
ADR-014 deferred WebSub for v1 as "complexity, external dependency".
Two existing problems matter here:
- Feeds are cached for `FEED_CACHE_SECONDS` (5 minutes) and nothing clears
the cache when a note changes. A hub fetching right after a change would
get stale content. Regular readers are also up to 5 minutes behind.
- The RSS feed is served at `/feed.rss` and `/feed.xml` (legacy alias), and
calls itself `/feed.xml`.
## Decision
StarPunk runs its own hub, **for its own topics only**, at `/websub`.
`WEBSUB_HUB_URL` can point at an external hub instead; then StarPunk
advertises that hub, pings it on changes, and its own `/websub` is off.
### 1. Topics
Absolute URLs under `SITE_URL`:
| Topic | Content type |
|---|---|
| `/` (h-feed) | text/html |
| `/feed.atom` | application/atom+xml |
| `/feed.rss` | application/rss+xml |
| `/feed.json` | application/feed+json |
| `/likes` | text/html |
| `/tag/<name>` | text/html |
`/feed.xml` and the negotiated `/feed` advertise the topic of the format
they serve as `rel=self` (`/feed.rss`, `/feed.atom`, `/feed.json`). The
RSS feed's own self link changes from `/feed.xml` to `/feed.rss`.
### 2. Discovery (publisher, spec 4)
Every topic response carries `Link: <hub>; rel="hub"` and exactly one
`Link: <topic>; rel="self"`. Embedded links as well: `<link rel="hub">` and
`<link rel="self">` in the HTML head of topic pages, `<link rel="hub">`
next to the existing self link in Atom, `<atom:link rel="hub">` in RSS, and
`"hubs": [{"type": "WebSub", "url": ...}]` in JSON Feed.
### 3. What changed, and when
`websub.note_changed(note, old_tags=())` is called wherever
`syndication.note_changed` is, plus on create. It:
1. clears the feed cache (all formats), and
2. if the note is or was public, queues delivery for `/`, the three
feeds, `/likes` if it is a like, and each `/tag/<name>` for its tags
before and after the change.
Drafts that stay drafts change nothing.
### 4. Hub endpoint (spec 5)
`POST /websub`, `application/x-www-form-urlencoded`:
- `hub.mode` subscribe or unsubscribe; `hub.topic` must be one of the
topics above (after normalising), else 400 with a plain-text reason;
`hub.callback` must be an absolute http(s) URL that passes the SSRF
check; `hub.secret` under 200 bytes; `hub.lease_seconds` clamped to
1 day to 30 days (default 10 days). Unknown parameters are ignored.
- Response: 202 Accepted, independent of what verification later finds.
- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions
per callback host and 1,000 in total; beyond that, 429.
### 5. Verification of intent (spec 5.3)
A background worker sends `GET <callback>` with the callback's query
string kept and `hub.mode`, `hub.topic`, `hub.challenge` (random, 32
bytes) and `hub.lease_seconds` appended. Success is a 2xx with the body
equal to the challenge; anything else, including redirects, fails and
leaves the subscription state unchanged. On success a subscribe becomes
active with `expires_at = now + lease`, and an unsubscribe deletes it.
A re-subscribe replaces the secret and lease only once verified.
### 6. Content distribution (spec 6, 7)
For each change, one delivery per active, unexpired subscription to the
topic. The worker renders the topic inside the app (no HTTP round trip
through Caddy), so subscribers get exactly what the public URL serves,
then `POST`s it to the callback with:
- `Content-Type` of the topic,
- `Link` headers with `rel=hub` and `rel=self`,
- `X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)>` if a secret was
given.
A 2xx is success. 410 Gone removes the subscription. Anything else is
retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a
subscription whose deliveries fail for 3 days is removed. Only the latest
content is sent: a queued delivery for a topic is replaced by a newer one.
### 7. Outbound safety
Callbacks are arbitrary URLs, so verification and delivery go through
`webmention_ssrf` (`safe_get`/`safe_post`): public addresses only, the
connection pinned to the checked address, no redirects, 10 s time limit,
1 MB response cap.
### 8. Data (migration 016)
- `websub_subscriptions(id, topic, callback, secret, lease_seconds,
state ['pending','active'], expires_at, created_at, verified_at,
last_success_at, last_error)`, unique on `(topic, callback)`.
- `websub_intents(id, topic, callback, mode, secret, lease_seconds,
attempts, next_attempt_at, created_at)`: verifications to do.
- `websub_deliveries(id, subscription_id, attempts, next_attempt_at,
last_error, created_at)`, unique on `subscription_id`.
### 9. Worker
The same pattern as the Webmention worker: one thread per gunicorn
process, a file lock so only one process works the queues, app context
around each poll. Expired subscriptions are pruned once a day.
### 10. Admin
The admin dashboard shows active subscribers per topic and failing
deliveries.
### 11. Configuration
`WEBSUB_ENABLED` (default true), `WEBSUB_HUB_URL` (default: built-in),
`WEBSUB_MAX_SUBSCRIPTIONS` (1000), `WEBSUB_MAX_PER_HOST` (25).
## Consequences
- Readers that support WebSub get posts and edits within seconds.
- Feeds are fresh right after a change for every reader, because the cache
is cleared.
- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF
protections, per-host and total limits, and rate limiting bound that.
- websub.rocks' hub tests use websub.rocks' own topics, which this hub
refuses by design. Testing uses unit tests of each spec section and a
real subscription from a reader after deploy.
- The RSS self link changes to `/feed.rss`. Readers keep whichever URL
they subscribed to; both keep working.
## Alternatives considered
- **Public hub only.** Least code, but a third-party dependency for a
self-hosted site. Still available through `WEBSUB_HUB_URL`.
- **A separate hub service** (another container). More to deploy and
keep updated, and few maintained hubs exist.
- **Accepting any topic** (a general-purpose hub). Not needed, and it
would let the hub be used to fetch and push arbitrary URLs.
## Amendments (v2.10.1, 2026-11-03)
An adversarial review of v2.10.0 against the Recommendation led to these
changes; where they differ, they replace the text above.
- **Worker robustness.** Each queued item is handled on its own: an
unexpected error drops (verification) or retries (delivery) that item
only. Before, a callback answering with a large page or containing a
control character stopped the worker for good.
- **Time limits.** `safe_get` and `safe_post` close the connection when
the whole exchange exceeds its time limit (10 s), not just a single read.
Verification replies over 4 KB fail; delivery responses are read up to
4 KB.
- **Leases are kept (spec 7).** A failing subscriber is not removed; its
deliveries are retried and then given up, and the next change is
delivered again. Only lease expiry, 410 Gone or unsubscribing end a
subscription. DNS failures are retried. A topic that renders 404/410
ends its subscriptions with a `hub.mode=denied` notice; other render
errors are retried.
- **Photos.** A note with photos is announced once they are attached
(`attach_media_to_note` notifies), and the feed cache key and ETag
include `content_version`, so photo and caption changes reach feeds and
subscribers. ETags no longer depend on the feed's build time.
- **Subscription requests (spec 5.1).** Unreserved percent-encodings are
decoded; the Content-Type check is case-insensitive; `hub.secret` and
`hub.lease_seconds` are ignored for unsubscriptions; an empty secret is
refused; verification echoes `hub.topic` exactly as requested
(`requested_topic`); unsubscribing works while an external hub is
configured. Callbacks must be printable, at most 2,048 characters, not on
this site, on port 80, 443 or 1024+, and are stored with an IDNA host.
- **Limits** count pending requests as well as subscriptions, per callback
*address* (so subdomains on one server share a limit), with at most 200
pending requests. Rate limiting needs `TRUSTED_PROXY=true` behind Caddy.
- **Races.** Each request has a `request_id`; verifying an older request
doesn't consume a newer one (migration 017).
- **Less work.** A topic is rendered once per worker pass; likes only
update `/likes` and tag pages.
- Corrections: there is no `state` column (a subscription row exists only
once verified); expired leases are pruned hourly.
+55
View File
@@ -0,0 +1,55 @@
-- Migration 016: WebSub hub (v2.10.0, ADR-071)
-- Active and pending subscriptions to this site's topics
CREATE TABLE IF NOT EXISTS websub_subscriptions (
id INTEGER PRIMARY KEY AUTOINCREMENT,
topic TEXT NOT NULL,
callback TEXT NOT NULL,
secret TEXT,
lease_seconds INTEGER NOT NULL,
expires_at TIMESTAMP NOT NULL,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
verified_at TIMESTAMP,
last_success_at TIMESTAMP,
last_error TEXT,
UNIQUE(topic, callback)
);
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_topic ON websub_subscriptions(topic);
-- Subscribe/unsubscribe requests waiting for verification of intent
CREATE TABLE IF NOT EXISTS websub_intents (
id INTEGER PRIMARY KEY AUTOINCREMENT,
topic TEXT NOT NULL,
callback TEXT NOT NULL,
mode TEXT NOT NULL CHECK (mode IN ('subscribe', 'unsubscribe')),
secret TEXT,
lease_seconds INTEGER,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
UNIQUE(topic, callback)
);
-- Content to push: one row per subscription (built-in hub) or per topic
-- (ping to an external hub, subscription_id NULL)
CREATE TABLE IF NOT EXISTS websub_deliveries (
id INTEGER PRIMARY KEY AUTOINCREMENT,
subscription_id INTEGER REFERENCES websub_subscriptions(id) ON DELETE CASCADE,
topic TEXT NOT NULL,
attempts INTEGER NOT NULL DEFAULT 0,
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
last_error TEXT,
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_subscription
ON websub_deliveries(subscription_id) WHERE subscription_id IS NOT NULL;
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_ping
ON websub_deliveries(topic) WHERE subscription_id IS NULL;
-- Site-wide values. content_version changes whenever a public note does, so
-- every gunicorn process knows its cached feed note list is stale.
CREATE TABLE IF NOT EXISTS site_state (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
INSERT OR IGNORE INTO site_state (key, value) VALUES ('content_version', '0');
+12
View File
@@ -0,0 +1,12 @@
-- Migration 017: WebSub hardening (v2.10.1)
-- request_id: each subscription request gets its own id, so verifying an
-- older request can't consume a newer one for the same topic and callback.
-- requested_topic: the hub.topic exactly as sent; verification echoes it.
-- callback_ip: address the callback resolved to, for per-address limits.
ALTER TABLE websub_intents ADD COLUMN request_id TEXT;
ALTER TABLE websub_intents ADD COLUMN requested_topic TEXT;
ALTER TABLE websub_intents ADD COLUMN callback_ip TEXT;
ALTER TABLE websub_subscriptions ADD COLUMN callback_ip TEXT;
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_ip ON websub_subscriptions(callback_ip);
CREATE INDEX IF NOT EXISTS idx_websub_intents_ip ON websub_intents(callback_ip);
+15 -8
View File
@@ -309,6 +309,11 @@ def create_app(config=None):
response.headers.add('Link', '</webmention>; rel="webmention"')
return response
# WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0)
from starpunk.websub import add_discovery_headers, template_links
app.after_request(add_discovery_headers)
app.add_template_global(template_links, 'websub_links')
# Register centralized error handlers
from starpunk.errors import register_error_handlers
@@ -322,12 +327,8 @@ def create_app(config=None):
memory_monitor.start()
app.memory_monitor = memory_monitor
app.logger.info(f"Memory monitor started (interval={memory_monitor.interval}s)")
# Register cleanup handler
@app.teardown_appcontext
def cleanup_memory_monitor(error=None):
if hasattr(app, 'memory_monitor') and app.memory_monitor.is_alive():
app.memory_monitor.stop()
# No teardown_appcontext stop: that ran after every request and stopped
# the monitor straight away (v2.10.0). It is a daemon thread.
# Start webmention worker thread (v1.6.0 Phase 2)
# Per background-worker.md: Skip in test mode
@@ -350,6 +351,12 @@ def create_app(config=None):
f"batch_size={webmention_worker.batch_size})"
)
# Start WebSub worker (v2.10.0)
from starpunk.websub_worker import start_worker
app.websub_worker = start_worker(app)
if app.websub_worker:
app.logger.info("WebSub worker started")
# Start syndication worker when a target is configured (v2.1.0)
from starpunk.syndication import SyndicationWorker, targets_configured
if targets_configured(app.config) and not app.config.get("TESTING", False):
@@ -463,5 +470,5 @@ def create_app(config=None):
# Package version (Semantic Versioning 2.0.0)
# See docs/standards/versioning-strategy.md for details
__version__ = "2.9.0"
__version_info__ = (2, 9, 0)
__version__ = "2.10.1"
__version_info__ = (2, 10, 1)
+7
View File
@@ -111,6 +111,13 @@ def load_config(app, config_override=None):
# Webmention configuration (v1.6.0)
app.config["WEBMENTION_ENABLED"] = os.getenv("WEBMENTION_ENABLED", "true").lower() == "true"
# WebSub (v2.10.0, ADR-071): built-in hub unless WEBSUB_HUB_URL names another
app.config["WEBSUB_ENABLED"] = os.getenv("WEBSUB_ENABLED", "true").lower() == "true"
app.config["WEBSUB_HUB_URL"] = os.getenv("WEBSUB_HUB_URL", "").strip()
app.config["WEBSUB_MAX_SUBSCRIPTIONS"] = int(os.getenv("WEBSUB_MAX_SUBSCRIPTIONS", "1000"))
app.config["WEBSUB_MAX_PER_HOST"] = int(os.getenv("WEBSUB_MAX_PER_HOST", "25"))
app.config["WEBSUB_POLL_INTERVAL"] = int(os.getenv("WEBSUB_POLL_INTERVAL", "5"))
app.config["WEBMENTION_WORKER_ENABLED"] = os.getenv("WEBMENTION_WORKER_ENABLED", "true").lower() == "true"
app.config["WEBMENTION_POLL_INTERVAL"] = int(os.getenv("WEBMENTION_POLL_INTERVAL", "10"))
app.config["WEBMENTION_ERROR_BACKOFF"] = int(os.getenv("WEBMENTION_ERROR_BACKOFF", "60"))
+5 -1
View File
@@ -169,7 +169,11 @@ class FeedCache:
- Adds new entry or updates existing entry
"""
cache_key = self._generate_cache_key(format_name, notes_checksum)
etag = self._generate_etag(content)
# From what the feed is made of, not its bytes: the bytes include a
# build time, so each server process made a different ETag and
# conditional requests rarely matched (v2.10.1). Weak ETag: same
# notes, semantically the same feed.
etag = self._generate_etag(f"{format_name}:{notes_checksum}")
timestamp = time.time()
# Evict if cache is full
+2 -2
View File
@@ -93,7 +93,7 @@ def generate_rss(
fg.language("en")
# Add self-link for feed discovery (Atom namespace)
fg.link(href=f"{site_url}/feed.xml", rel="self", type="application/rss+xml")
fg.link(href=f"{site_url}/feed.rss", rel="self", type="application/rss+xml")
# Set last build date to now
fg.lastBuildDate(datetime.now(timezone.utc))
@@ -246,7 +246,7 @@ def generate_rss_streaming(
yield f" <description>{_escape_xml(site_description or site_name)}</description>\n"
yield " <language>en</language>\n"
yield f" <lastBuildDate>{last_build}</lastBuildDate>\n"
yield f' <atom:link href="{_escape_xml(site_url)}/feed.xml" rel="self" type="application/rss+xml"/>\n'
yield f' <atom:link href="{_escape_xml(site_url)}/feed.rss" rel="self" type="application/rss+xml"/>\n'
# Yield items (newest first)
# Notes from database are already in DESC order (newest first)
+9
View File
@@ -891,6 +891,15 @@ def attach_media_to_note(note_id: int, media_ids: List[int], captions: List[str]
db.rollback()
raise
# Feeds and pages show the photos: refresh caches and tell WebSub (v2.10.1)
try:
from starpunk.notes import _websub_changed, get_note
note = get_note(id=note_id, load_content=False)
if note is not None:
_websub_changed(note, was_public=note.published)
except Exception as e:
current_app.logger.warning(f"Failed to queue WebSub update for note {note_id}: {e}")
def get_note_media(note_id: int) -> List[Dict]:
"""
+1 -1
View File
@@ -760,7 +760,7 @@ def handle_update(data: dict, token_info: dict):
if "photo" in touched:
from starpunk.media import attach_media_to_note
attach_media_to_note(note.id, media_ids, captions)
note_changed(note.id)
note_changed(note.id) # attach_media_to_note tells WebSub
except InvalidNoteDataError as e:
raise MicropubValidationError(str(e))
except MicropubError:
+36 -1
View File
@@ -374,9 +374,31 @@ def create_note(
# Reply context fetch failure should not prevent note creation
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
# A note with photos is announced when they're attached
# (attach_media_to_note), so subscribers never get it without them
if not has_media:
_websub_changed(note)
return note
def _tag_names(note_id: int) -> list[str]:
"""A note's tag names, for WebSub tag topics (best effort)"""
try:
from starpunk.tags import get_note_tags
return [t["name"] for t in get_note_tags(note_id)]
except Exception:
return []
def _websub_changed(note, was_public: bool = False, old_tags=()) -> None:
"""Clear cached feeds and queue WebSub deliveries (v2.10.0, best effort)"""
try:
from starpunk import websub
websub.note_changed(note, was_public=was_public, old_tags=old_tags)
except Exception as e:
current_app.logger.warning(f"Failed to queue WebSub update for note {note.slug}: {e}")
def get_note(
slug: Optional[str] = None, id: Optional[int] = None, load_content: bool = True
) -> Optional[Note]:
@@ -705,6 +727,7 @@ def update_note(
if existing_note is None:
identifier = slug if slug is not None else id
raise NoteNotFoundError(identifier)
old_tag_names = _tag_names(existing_note.id) # for WebSub tag topics (v2.10.0)
if empty_content and not (existing_note.reacts_to or has_media):
raise InvalidNoteDataError(
@@ -896,6 +919,8 @@ def update_note(
except Exception as e:
current_app.logger.warning(f"Failed to queue syndication update: {e}")
_websub_changed(updated_note, was_public=existing_note.published, old_tags=old_tag_names)
# 10. RETURN UPDATED NOTE
return updated_note
@@ -995,6 +1020,10 @@ def delete_note(
# For idempotency, don't raise error - just return
return
# What feeds showed before, for WebSub (v2.10.0)
was_public = bool(existing_note.published) and not getattr(existing_note, "deleted_at", None)
deleted_tags = _tag_names(existing_note.id)
# 4. SETUP
data_dir = Path(current_app.config["DATA_PATH"])
note_path = data_dir / existing_note.file_path
@@ -1081,6 +1110,10 @@ def delete_note(
except Exception as e:
current_app.logger.warning(f"Failed to queue syndication delete: {e}")
# Feeds and pages no longer show it (v2.10.0)
object.__setattr__(existing_note, "published", False)
_websub_changed(existing_note, was_public=was_public, old_tags=deleted_tags)
# 6. RETURN (no value)
return None
@@ -1137,4 +1170,6 @@ def undelete_note(slug: str) -> Note:
except Exception as e:
current_app.logger.warning(f"Failed to queue webmentions after undelete: {e}")
return get_note(slug=slug)
restored = get_note(slug=slug)
_websub_changed(restored)
return restored
+4 -1
View File
@@ -7,7 +7,7 @@ admin, auth, and (conditionally) dev auth routes.
from flask import Flask
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention, websub
def register_routes(app: Flask) -> None:
@@ -44,6 +44,9 @@ def register_routes(app: Flask) -> None:
if app.config.get("WEBMENTION_ENABLED", True):
app.register_blueprint(webmention.bp)
# WebSub hub (v2.10.0); refuses requests when disabled or using another hub
app.register_blueprint(websub.bp)
# Conditionally register dev auth routes
if app.config.get("DEV_MODE"):
app.logger.warning(
+21 -1
View File
@@ -72,12 +72,23 @@ def dashboard():
except Exception:
pass
# WebSub subscribers (v2.10.0)
websub_stats = None
if current_app.config.get('WEBSUB_ENABLED', True):
try:
from starpunk import websub
from starpunk.database import get_db
websub_stats = websub.stats(get_db(current_app))
except Exception:
pass
return render_template(
"admin/dashboard.html",
notes=notes,
page=page,
total_pages=total_pages,
webmention_pending_count=webmention_pending_count,
websub_stats=websub_stats,
user_me=g.me
)
@@ -139,13 +150,17 @@ def create_note_submit():
# Parse tags (v1.3.0 Phase 3)
tags = parse_tag_input(tags_input)
# With photos, WebSub hears about the note once they're attached (v2.10.1)
has_uploads = any(f.filename for f in request.files.getlist('media_files'))
try:
# Create note first (per Q4)
note = create_note(
content,
published=published,
custom_slug=custom_slug if custom_slug else None,
tags=tags if tags else None
tags=tags if tags else None,
has_media=has_uploads,
)
# Handle media uploads (v1.2.0 Phase 3)
@@ -184,6 +199,11 @@ def create_note_submit():
if errors:
flash(f"Note created, but some images failed: {'; '.join(errors)}", "warning")
if not media_ids:
# No photo made it; announce the note as it is (v2.10.1)
from starpunk.notes import _websub_changed
_websub_changed(note)
# Crosspost after media is attached so images go out too (v2.1.0)
if "syndication_form" in request.form:
from starpunk.syndication import select_targets
+24 -9
View File
@@ -11,6 +11,7 @@ from datetime import datetime, timedelta
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
from starpunk.notes import list_notes, get_note
from starpunk.websub import embed_hub
from starpunk.feed import generate_feed_streaming # Legacy RSS
from starpunk.feeds import (
generate_rss,
@@ -55,8 +56,14 @@ def _get_cached_notes():
cache_duration = timedelta(seconds=cache_seconds)
now = datetime.utcnow()
# The cached list is stale once any public note changed, in any process
# (v2.10.0): every change bumps content_version in the database
from starpunk.database import get_db
from starpunk.websub import get_content_version
version = (str(current_app.config["DATABASE_PATH"]), get_content_version(get_db(current_app)))
# Check if note list cache is valid
if _feed_cache["notes"] and _feed_cache["timestamp"]:
if _feed_cache["notes"] and _feed_cache["timestamp"] and _feed_cache.get("version") == version:
cache_age = now - _feed_cache["timestamp"]
if cache_age < cache_duration:
# Use cached note list
@@ -82,6 +89,7 @@ def _get_cached_notes():
_feed_cache["notes"] = notes
_feed_cache["timestamp"] = now
_feed_cache["version"] = version
return notes
@@ -122,6 +130,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
notes=notes,
limit=max_items,
)
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
# Create response with proper Content-Type including charset
mime_type = get_mime_type(format_name)
@@ -132,7 +141,12 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
# Caching enabled - use FeedCache
feed_cache = get_cache()
notes_checksum = feed_cache.generate_notes_checksum(notes)
# The content version covers changes the note list doesn't show, such as
# photos attached or captions edited (v2.10.1)
from starpunk.database import get_db
from starpunk.websub import get_content_version
notes_checksum = (feed_cache.generate_notes_checksum(notes)
+ ":" + get_content_version(get_db(current_app)))
# Check If-None-Match header for conditional requests
if_none_match = request.headers.get('If-None-Match')
@@ -170,6 +184,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
notes=notes,
limit=max_items,
)
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
# Store in cache and get ETag
etag = feed_cache.set(format_name, content, notes_checksum)
@@ -498,15 +513,15 @@ def feed():
)
# Route to appropriate generator
if format_name == 'rss':
return feed_rss()
elif format_name == 'atom':
return feed_atom()
if format_name == 'atom':
response = feed_atom()
elif format_name == 'json':
return feed_json()
response = feed_json()
else:
# Shouldn't reach here, but be defensive
return feed_rss()
response = feed_rss()
# The format (and its rel=self) depends on Accept (v2.10.1)
response.headers.add('Vary', 'Accept')
return response
@bp.route("/feed.rss")
+57
View File
@@ -0,0 +1,57 @@
"""
WebSub hub endpoint (v2.10.0, ADR-071)
POST /websub with hub.mode subscribe/unsubscribe, hub.topic, hub.callback,
optional hub.secret and hub.lease_seconds. Answers 202 and verifies later
(spec 5.1); errors are 4xx with a plain-text reason.
"""
from flask import Blueprint, current_app, request
from starpunk.database import get_db
from starpunk.rate_limiter import RateLimiter
from starpunk import websub
bp = Blueprint("websub", __name__)
# Subscription requests per client IP per minute
# (per gunicorn process; needs TRUSTED_PROXY=true behind a reverse proxy,
# or every client shares the proxy's address)
_limiter = RateLimiter(limit=30, window_seconds=60)
_requests = 0
def _text(message: str, status: int):
return message + "\n", status, {"Content-Type": "text/plain; charset=utf-8"}
@bp.route("/websub", methods=["POST"])
def hub():
if not websub.enabled():
return _text("WebSub is not enabled on this site", 404)
from starpunk.routes.webmention import get_client_ip
global _requests
_requests += 1
if _requests % 500 == 0:
_limiter.cleanup() # forget idle clients so memory doesn't grow
if not _limiter.is_allowed(get_client_ip()):
return _text("Too many requests", 429)
# Media types are case-insensitive (v2.10.1)
if request.mimetype != "application/x-www-form-urlencoded":
return _text("Send an application/x-www-form-urlencoded body", 400)
try:
websub.accept_subscription_request(request.form, get_db(current_app))
except websub.SubscriptionError as e:
return _text(str(e), e.status)
return _text("Accepted; the subscription will be verified", 202)
@bp.route("/websub", methods=["GET"])
def hub_info():
return _text(
"WebSub hub for this site's own feeds. Subscribe with a POST "
"(https://www.w3.org/TR/websub/).", 200)
+73 -11
View File
@@ -16,8 +16,10 @@ References:
import ipaddress
import socket
import threading
import time
from dataclasses import dataclass
from typing import Optional
from urllib.parse import urljoin, urlparse
import httpx
@@ -39,6 +41,11 @@ class SSRFBlockedError(Exception):
pass
class ResolutionError(ValueError):
"""The host name could not be resolved (may be temporary; v2.10.1)"""
pass
class FetchTooLargeError(Exception):
"""Raised when a response is larger than the limit"""
pass
@@ -131,7 +138,7 @@ def validate_url_for_fetch(url: str) -> str:
try:
return safe_resolve(parsed.hostname)
except socket.gaierror as e:
raise ValueError(f"Could not resolve {parsed.hostname}: {e}")
raise ResolutionError(f"Could not resolve {parsed.hostname}: {e}")
def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
@@ -157,18 +164,36 @@ def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
)
def safe_post(url: str, data: dict, headers: dict, timeout: float = TIMEOUT) -> httpx.Response:
class _Watchdog:
"""
POST form data to a checked, pinned address. Redirects are not followed.
Close a client when its deadline passes (v2.10.1)
Raises:
SSRFBlockedError, ValueError: URL is not allowed
httpx.HTTPError: Network errors
httpx timeouts apply per read, so a server sending one byte at a time
could hold a request for hours; closing the client ends it.
"""
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
response = client.send(_pinned_request(client, 'POST', url, headers, data=data))
response.read()
return response
def __init__(self, client: httpx.Client, seconds: float):
self.fired = False
def fire():
self.fired = True
try:
client.close()
except Exception:
pass
self._timer = threading.Timer(max(seconds, 0.01), fire)
self._timer.daemon = True
def __enter__(self):
self._timer.start()
return self
def __exit__(self, exc_type, exc, tb):
self._timer.cancel()
if self.fired and exc_type is not None and not isinstance(exc, httpx.TimeoutException):
raise httpx.TimeoutException("Exceeded time limit") from exc
return False
@dataclass
@@ -188,6 +213,42 @@ class FetchResult:
return self.content.decode('utf-8', errors='replace')
def safe_post(url: str, data: Optional[dict] = None, headers: Optional[dict] = None,
timeout: float = TIMEOUT, content: Optional[bytes] = None,
max_bytes: int = 4096) -> FetchResult:
"""
POST form data (or a raw body) to a checked, pinned address.
Redirects are not followed. `timeout` bounds the whole exchange, so a
server trickling its response can't hold the caller (v2.10.1). Only the
first max_bytes of the response are read (callers need the status and
maybe an error message).
Raises:
SSRFBlockedError, ValueError: URL is not allowed
httpx.HTTPError: Network errors
"""
kwargs = {'content': content} if content is not None else {'data': data}
deadline = time.monotonic() + timeout
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client, \
_Watchdog(client, timeout):
request = _pinned_request(client, 'POST', url, headers or {}, **kwargs)
response = client.send(request, stream=True)
try:
body = b''
for chunk in response.iter_bytes(chunk_size=1024):
body += chunk
if len(body) >= max_bytes:
body = body[:max_bytes]
break
if time.monotonic() > deadline:
raise httpx.TimeoutException("Response exceeded time limit")
return FetchResult(url=url, status_code=response.status_code, headers=response.headers,
content=body, encoding=response.charset_encoding or 'utf-8')
finally:
response.close()
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
"""
@@ -210,7 +271,8 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
"""
deadline = time.monotonic() + timeout
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client:
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client, \
_Watchdog(client, timeout):
for _ in range(max_redirects + 1):
remaining = deadline - time.monotonic()
if remaining <= 0:
+443
View File
@@ -0,0 +1,443 @@
"""
WebSub publishing and the built-in hub (v2.10.0, ADR-071)
Publisher: topic responses advertise the hub and their own URL; a change to a
public note clears cached feeds and queues delivery of every affected topic.
Hub: subscriptions to this site's own topics only. Requests are accepted
here (202) and verified, delivered and retried by the WebSub worker.
References:
- W3C WebSub: https://www.w3.org/TR/websub/
- ADR-071: WebSub publishing with a built-in hub
"""
import json
import secrets
from typing import Iterable, Optional
from urllib.parse import urlparse
from flask import current_app, request
# Topic endpoints and their fixed paths (tag pages are added per tag)
FEED_TOPICS = {
"public.feed_atom": "/feed.atom",
"public.feed_rss": "/feed.rss",
"public.feed_xml_legacy": "/feed.rss", # legacy alias of the RSS topic
"public.feed_json": "/feed.json",
}
PAGE_TOPICS = {"public.index": "/", "public.likes": "/likes"}
NEGOTIATED = { # /feed: the topic of whichever format was served
"application/atom+xml": "/feed.atom",
"application/rss+xml": "/feed.rss",
"application/feed+json": "/feed.json",
"application/json": "/feed.json",
}
LEASE_DEFAULT = 10 * 24 * 3600
LEASE_MIN = 24 * 3600
LEASE_MAX = 30 * 24 * 3600
MAX_SECRET_BYTES = 200
class SubscriptionError(Exception):
"""A subscription request the hub refuses (4xx with a plain-text reason)"""
def __init__(self, message: str, status: int = 400):
super().__init__(message)
self.status = status
# --- Configuration -----------------------------------------------------------
def enabled() -> bool:
return bool(current_app.config.get("WEBSUB_ENABLED", True))
def external_hub() -> Optional[str]:
"""URL of an external hub, if one is configured instead of the built-in one"""
return current_app.config.get("WEBSUB_HUB_URL") or None
def hub_url() -> str:
# Built from SITE_URL so the worker can use it outside a request
return external_hub() or site_url() + "/websub"
def site_url() -> str:
return current_app.config["SITE_URL"].rstrip("/")
# --- Topics ------------------------------------------------------------------
def topic_url(path: str) -> str:
return site_url() + path
def tag_topic(tag_name: str) -> str:
from urllib.parse import quote
return f"{site_url()}/tag/{quote(tag_name, safe='')}"
def topic_for_request(response=None) -> Optional[str]:
"""The topic URL the current request serves, or None if it isn't a topic"""
endpoint = request.endpoint
if endpoint in PAGE_TOPICS:
return topic_url(PAGE_TOPICS[endpoint])
if endpoint in FEED_TOPICS:
return topic_url(FEED_TOPICS[endpoint])
if endpoint == "public.feed" and response is not None:
mime = (response.mimetype or "").lower()
return topic_url(NEGOTIATED[mime]) if mime in NEGOTIATED else None
if endpoint == "public.tag":
from starpunk.tags import normalize_tag
tag = (request.view_args or {}).get("tag", "")
return tag_topic(normalize_tag(tag)[0]) if tag else None
return None
_UNRESERVED = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")
def decode_unreserved(value: str) -> str:
"""
Decode percent-encoded unreserved characters (%66 -> f), leaving the rest
Spec 5.1.1: hubs MUST always decode non-reserved characters in URL
parameters, so /%66eed.atom and /feed.atom are the same topic.
"""
out, i = [], 0
while i < len(value):
if value[i] == "%" and i + 3 <= len(value):
try:
char = chr(int(value[i + 1:i + 3], 16))
except ValueError:
char = None
if char is not None and char in _UNRESERVED:
out.append(char)
i += 3
continue
out.append(value[i])
i += 1
return "".join(out)
def canonical_topic(url: str) -> Optional[str]:
"""
A subscription's hub.topic as one of this site's topics, or None
Unreserved percent-encodings are decoded and the scheme and host are
compared case-insensitively. Tag topics must name an existing tag.
"""
if not url:
return None
site = urlparse(site_url())
parsed = urlparse(decode_unreserved(url.strip()))
if (parsed.scheme.lower(), parsed.netloc.lower()) != (site.scheme, site.netloc.lower()):
return None
path = parsed.path or "/"
if parsed.query or parsed.fragment:
return None
if path in set(PAGE_TOPICS.values()) | set(FEED_TOPICS.values()):
return topic_url(path)
if path.startswith("/tag/"):
from urllib.parse import unquote
from starpunk.tags import get_tag_by_name, normalize_tag
name = normalize_tag(unquote(path[len("/tag/"):]))[0]
if name and get_tag_by_name(name):
return tag_topic(name)
return None
def link_header_values(topic: str) -> list[str]:
return [f'<{hub_url()}>; rel="hub"', f'<{topic}>; rel="self"']
def add_discovery_headers(response):
"""after_request: Link rel=hub and rel=self on topic responses (spec 4)"""
if not enabled() or response.status_code not in (200, 304):
return response
topic = topic_for_request(response)
if topic:
for value in link_header_values(topic):
response.headers.add("Link", value)
return response
def template_links() -> Optional[dict]:
"""For base.html: {'hub', 'self'} on topic pages"""
if not enabled() or not request:
return None
topic = topic_for_request()
return {"hub": hub_url(), "self": topic} if topic else None
def embed_hub(format_name: str, content: str) -> str:
"""Add the hub link inside a generated feed, next to its self link"""
if not enabled():
return content
hub = hub_url()
if format_name == "atom":
marker = '<link rel="self"'
return content.replace(marker, f'<link rel="hub" href="{_xml(hub)}"/>\n {marker}', 1)
if format_name == "rss":
marker = "<atom:link "
return content.replace(marker, f'<atom:link href="{_xml(hub)}" rel="hub"/>\n {marker}', 1)
if format_name == "json":
feed = json.loads(content)
feed["hubs"] = [{"type": "WebSub", "url": hub}]
return json.dumps(feed, ensure_ascii=False, indent=2)
return content
def _xml(value: str) -> str:
return (value.replace("&", "&amp;").replace("<", "&lt;")
.replace(">", "&gt;").replace('"', "&quot;"))
# --- Changes -------------------------------------------------------------------
def get_content_version(db) -> str:
row = db.execute("SELECT value FROM site_state WHERE key = 'content_version'").fetchone()
return row[0] if row else "0"
def bump_content_version(db) -> None:
db.execute(
"INSERT INTO site_state (key, value) VALUES ('content_version', '1') "
"ON CONFLICT(key) DO UPDATE SET value = CAST(value AS INTEGER) + 1"
)
def note_changed(note, was_public: bool = False, old_tags: Iterable[str] = ()) -> None:
"""
A note was created, edited, deleted, restored or (un)published
Clears cached feeds in every process and queues delivery of the topics
that show it. Changes that are never public (a draft edited) do nothing.
Args:
note: The note (after the change; for a delete, before it)
was_public: Whether it was public before the change
old_tags: Its tag names before the change
"""
from starpunk.database import get_db
is_public = bool(note.published) and not getattr(note, "deleted_at", None)
if not (is_public or was_public):
return
db = get_db(current_app)
bump_content_version(db)
db.commit()
if not enabled():
return
from starpunk.tags import get_note_tags
try:
tags = {t["name"] for t in get_note_tags(note.id)}
except Exception:
tags = set()
tags |= set(old_tags)
# Likes appear on /likes, not on the home page or in the feeds
if getattr(note, "like_of", None):
topics = [topic_url("/likes")]
else:
topics = [topic_url(p) for p in ("/", "/feed.atom", "/feed.rss", "/feed.json")]
topics += [tag_topic(name) for name in sorted(tags)]
queue_topics(db, topics)
def queue_topics(db, topics: Iterable[str]) -> int:
"""Queue delivery of topics: to their subscribers, or a ping to an external hub"""
topics = list(dict.fromkeys(topics))
if not topics:
return 0
if external_hub():
for topic in topics:
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NULL AND topic = ?", (topic,))
db.execute("INSERT INTO websub_deliveries (topic) VALUES (?)", (topic,))
db.commit()
return len(topics)
placeholders = ",".join("?" * len(topics))
rows = db.execute(
f"SELECT id, topic FROM websub_subscriptions WHERE topic IN ({placeholders}) "
"AND verified_at IS NOT NULL AND expires_at > datetime('now')",
topics,
).fetchall()
for sub_id, topic in rows:
# The newest content replaces anything still waiting (ADR-071 6)
db.execute("DELETE FROM websub_deliveries WHERE subscription_id = ?", (sub_id,))
db.execute("INSERT INTO websub_deliveries (subscription_id, topic) VALUES (?, ?)", (sub_id, topic))
db.commit()
return len(rows)
# --- Hub: subscription requests (spec 5.1) -------------------------------------
MAX_CALLBACK_LENGTH = 2048
MAX_PENDING_INTENTS = 200
ALLOWED_PORTS = {80, 443} # plus any port from 1024 up
def _clean_callback(callback: str) -> str:
"""
Check a hub.callback and return it with an ASCII (IDNA) host
Raises:
SubscriptionError: if it is not an acceptable callback URL
"""
import httpx
if not callback or len(callback) > MAX_CALLBACK_LENGTH:
raise SubscriptionError(f"hub.callback must be a URL of at most {MAX_CALLBACK_LENGTH} characters")
if any(ord(c) < 0x20 or ord(c) == 0x7f for c in callback) or " " in callback:
raise SubscriptionError("hub.callback contains spaces or control characters")
parsed = urlparse(callback)
if parsed.scheme.lower() not in ("http", "https") or not parsed.hostname:
raise SubscriptionError("hub.callback must be an absolute http(s) URL")
try:
port = parsed.port
except ValueError:
raise SubscriptionError("hub.callback has an invalid port")
if port is not None and port not in ALLOWED_PORTS and port < 1024:
raise SubscriptionError("hub.callback may not use that port")
try:
host = parsed.hostname.encode("idna").decode("ascii")
except UnicodeError:
raise SubscriptionError("hub.callback has an invalid host name")
netloc = host + (f":{port}" if port is not None else "")
callback = parsed._replace(scheme=parsed.scheme.lower(), netloc=netloc).geturl()
if host.lower() == (urlparse(site_url()).hostname or "").lower():
raise SubscriptionError("hub.callback may not be on this site")
try:
httpx.URL(callback)
except Exception:
raise SubscriptionError("hub.callback is not a valid URL")
return callback
def accept_subscription_request(form, db) -> None:
"""
Validate a subscription request and queue it for verification
Raises:
SubscriptionError: with the status and plain-text reason to return
"""
from starpunk.webmention_ssrf import SSRFBlockedError, validate_url_for_fetch
mode = (form.get("hub.mode") or "").strip()
if mode not in ("subscribe", "unsubscribe"):
raise SubscriptionError("hub.mode must be subscribe or unsubscribe")
if mode == "subscribe" and external_hub():
# Unsubscribes are still taken, so old subscribers can leave
raise SubscriptionError("This site uses another hub: " + external_hub(), 400)
callback = _clean_callback((form.get("hub.callback") or "").strip())
try:
callback_ip = validate_url_for_fetch(callback)
except (SSRFBlockedError, ValueError) as e:
raise SubscriptionError(f"hub.callback is not reachable: {e}")
requested_topic = (form.get("hub.topic") or "").strip()
topic = canonical_topic(requested_topic)
if not topic:
raise SubscriptionError("hub.topic is not a topic of this site; use the rel=self URL")
secret, lease = None, None
if mode == "subscribe":
# hub.secret and hub.lease_seconds MUST be ignored for unsubscriptions
secret = form.get("hub.secret")
if secret is not None:
if secret == "":
raise SubscriptionError("hub.secret may not be empty")
if len(secret.encode("utf-8")) >= MAX_SECRET_BYTES:
raise SubscriptionError("hub.secret must be less than 200 bytes")
lease = LEASE_DEFAULT
if form.get("hub.lease_seconds"):
try:
lease = int(form["hub.lease_seconds"])
except ValueError:
raise SubscriptionError("hub.lease_seconds must be a number")
lease = max(LEASE_MIN, min(lease, LEASE_MAX))
_check_limits(db, topic, callback, callback_ip)
# Each request gets its own id; the latest replaces any pending one
db.execute(
"""
INSERT INTO websub_intents
(topic, callback, mode, secret, lease_seconds, request_id, requested_topic, callback_ip)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(topic, callback) DO UPDATE SET
mode = excluded.mode, secret = excluded.secret,
lease_seconds = excluded.lease_seconds, request_id = excluded.request_id,
requested_topic = excluded.requested_topic, callback_ip = excluded.callback_ip,
attempts = 0, next_attempt_at = CURRENT_TIMESTAMP, created_at = CURRENT_TIMESTAMP
""",
(topic, callback, mode, secret, lease, secrets.token_hex(16), requested_topic, callback_ip),
)
db.commit()
def _check_limits(db, topic: str, callback: str, callback_ip: str) -> None:
"""
Subscription limits, counting pending requests too (ADR-071 4)
Per callback address rather than host name, so many subdomains on one
server share a limit. Renewing an existing subscription is always fine.
"""
if db.execute("SELECT 1 FROM websub_subscriptions WHERE topic = ? AND callback = ?",
(topic, callback)).fetchone():
return
pending = db.execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0]
if pending >= MAX_PENDING_INTENTS:
raise SubscriptionError("Too many subscription requests waiting; try again later", 429)
total = db.execute(
"""
SELECT COUNT(*) FROM (
SELECT topic, callback FROM websub_subscriptions
UNION
SELECT topic, callback FROM websub_intents WHERE mode = 'subscribe'
)
"""
).fetchone()[0]
if total >= current_app.config.get("WEBSUB_MAX_SUBSCRIPTIONS", 1000):
raise SubscriptionError("This hub has reached its subscription limit", 429)
per_address = db.execute(
"""
SELECT COUNT(*) FROM (
SELECT topic, callback FROM websub_subscriptions WHERE callback_ip = ?
UNION
SELECT topic, callback FROM websub_intents WHERE mode = 'subscribe' AND callback_ip = ?
)
""",
(callback_ip, callback_ip),
).fetchone()[0]
if per_address >= current_app.config.get("WEBSUB_MAX_PER_HOST", 25):
raise SubscriptionError("Too many subscriptions for that callback's server", 429)
def stats(db) -> dict:
"""Active subscribers per topic and deliveries being retried (admin)"""
by_topic = db.execute(
"SELECT topic, COUNT(*) FROM websub_subscriptions WHERE expires_at > datetime('now') "
"GROUP BY topic ORDER BY COUNT(*) DESC, topic"
).fetchall()
retrying = db.execute(
"SELECT COUNT(*) FROM websub_deliveries WHERE attempts > 0"
).fetchone()[0]
return {"by_topic": [(t, n) for t, n in by_topic], "retrying": retrying,
"external": external_hub()}
def new_challenge() -> str:
return secrets.token_urlsafe(32)
+362
View File
@@ -0,0 +1,362 @@
"""
WebSub worker: verification of intent and content distribution (v2.10.0)
Runs like the Webmention worker: a daemon thread in every gunicorn process,
but only the process holding the file lock works the queues.
- Intents (spec 5.3): GET the callback with a challenge; a 2xx echoing it
verifies. Anything else (including redirects) leaves the subscription as
it was. Network errors, DNS failures and 5xx are retried a few times.
- Deliveries (spec 7): render the topic in-app and POST it with
Link rel=hub/self and, if there is a secret, X-Hub-Signature (sha256).
Failures are retried; the subscription stays until its lease ends.
- External hub: a delivery without a subscription is a publish ping.
v2.10.1: every queue item is handled on its own, so one bad item can't stop
the worker; all outbound requests have a hard time limit.
References:
- https://www.w3.org/TR/websub/
- ADR-071
"""
import fcntl
import hashlib
import hmac
import logging
import threading
from datetime import datetime, timedelta
from pathlib import Path
from typing import Optional
from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit
import httpx
logger = logging.getLogger(__name__)
DELIVERY_BACKOFF = [60, 300, 1800, 7200, 21600] # then give up on this delivery
INTENT_BACKOFF = [60, 300, 900]
PRUNE_EVERY = timedelta(hours=1)
BATCH = 10
VERIFY_MAX_BYTES = 4096 # a challenge reply is short; larger means not verified
def callback_with_params(callback: str, params: dict) -> str:
"""Append parameters to the callback, keeping its query string (spec 5.3)"""
parts = urlsplit(callback)
query = parts.query + ("&" if parts.query else "") + urlencode(params)
return urlunsplit((parts.scheme, parts.netloc, parts.path, query, parts.fragment))
def signature(secret: str, body: bytes) -> str:
return "sha256=" + hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
def _now() -> str:
return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
def _later(seconds: int) -> str:
return (datetime.utcnow() + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S")
class WebSubWorker(threading.Thread):
"""Daemon thread that verifies subscriptions and delivers content"""
def __init__(self, app, poll_interval: int = 5):
super().__init__(daemon=True, name="websub-worker")
self.app = app
self.poll_interval = poll_interval
self._stop_event = threading.Event()
self._lock_file = None
self._last_prune = None
# --- thread plumbing ----------------------------------------------------
def run(self):
logger.info("WebSub worker started")
while not self._stop_event.is_set():
if not self._acquire_lock():
self._stop_event.wait(self.poll_interval)
continue
try:
with self.app.app_context():
done = self.process_once()
except Exception as e:
logger.error(f"WebSub worker error: {e}", exc_info=True)
done = 0
if not done:
self._stop_event.wait(self.poll_interval)
self._release_lock()
def stop(self):
self._stop_event.set()
def _acquire_lock(self) -> bool:
if self._lock_file:
return True
path = Path(self.app.config["DATABASE_PATH"]).parent / "websub-worker.lock"
lock_file = open(path, "w")
try:
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError:
lock_file.close()
return False
self._lock_file = lock_file
logger.info("WebSub worker holds the queue lock")
return True
def _release_lock(self):
if self._lock_file:
self._lock_file.close()
self._lock_file = None
# --- one pass (also called directly by tests) ---------------------------
def process_once(self) -> int:
from starpunk.database import get_db
db = get_db(self.app)
done = 0
rendered = {} # topic -> (status, content type, body), once per pass
for row in db.execute(
"SELECT * FROM websub_intents WHERE next_attempt_at <= ? ORDER BY next_attempt_at, id LIMIT ?",
(_now(), BATCH),
).fetchall():
self._guarded(db, row, "verification", self.verify, self._intent_crashed)
done += 1
for row in db.execute(
"SELECT * FROM websub_deliveries WHERE next_attempt_at <= ? ORDER BY next_attempt_at, id LIMIT ?",
(_now(), BATCH),
).fetchall():
self._guarded(db, row, "delivery", lambda d, r: self.deliver(d, r, rendered), self._retry_or_drop)
done += 1
if self._last_prune is None or datetime.utcnow() - self._last_prune > PRUNE_EVERY:
self.prune(db)
return done
def _guarded(self, db, row, what, handler, on_crash) -> None:
"""Handle one queue item; an unexpected error affects only that item"""
try:
handler(db, row)
except Exception as e:
logger.error(f"WebSub {what} {row['id']} failed unexpectedly: {e}", exc_info=True)
try:
db.rollback()
on_crash(db, row, f"{type(e).__name__}: {e}"[:200])
db.commit()
except Exception as cleanup:
logger.error(f"WebSub could not record the failure: {cleanup}")
def _intent_crashed(self, db, intent, error) -> None:
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
(intent["id"], intent["request_id"]))
# --- verification of intent (spec 5.3) ----------------------------------
def verify(self, db, intent) -> None:
from starpunk.websub import new_challenge
from starpunk.webmention_ssrf import (
FetchTooLargeError, ResolutionError, SSRFBlockedError, safe_get, user_agent,
)
challenge = new_challenge()
# The topic exactly as the subscriber sent it (spec 5.3)
params = {"hub.mode": intent["mode"],
"hub.topic": intent["requested_topic"] or intent["topic"],
"hub.challenge": challenge}
if intent["mode"] == "subscribe":
params["hub.lease_seconds"] = intent["lease_seconds"]
url = callback_with_params(intent["callback"], params)
retry = False
try:
result = safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
max_redirects=0, max_bytes=VERIFY_MAX_BYTES)
# Exact match; a trailing newline from a framework is tolerated
verified = 200 <= result.status_code < 300 and result.text.rstrip("\r\n") == challenge
retry = result.status_code >= 500
error = None if verified else f"HTTP {result.status_code}" + (
"" if result.status_code >= 300 else ": challenge not echoed")
except ResolutionError as e:
verified, retry, error = False, True, str(e)
except (SSRFBlockedError, ValueError) as e:
verified, error = False, f"Blocked: {e}"
except FetchTooLargeError:
verified, error = False, "Reply too large to be a challenge"
except httpx.TooManyRedirects:
verified, error = False, "Redirected (not allowed)"
except httpx.HTTPError as e:
verified, retry, error = False, True, (str(e) or type(e).__name__)[:200]
# Only act if no newer request replaced this one meanwhile
current = db.execute("SELECT request_id FROM websub_intents WHERE id = ?", (intent["id"],)).fetchone()
if current is None or current[0] != intent["request_id"]:
return
if verified:
if intent["mode"] == "subscribe":
db.execute(
"""
INSERT INTO websub_subscriptions
(topic, callback, secret, lease_seconds, expires_at, verified_at, callback_ip)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(topic, callback) DO UPDATE SET
secret = excluded.secret, lease_seconds = excluded.lease_seconds,
expires_at = excluded.expires_at, verified_at = excluded.verified_at,
callback_ip = excluded.callback_ip, last_error = NULL
""",
(intent["topic"], intent["callback"], intent["secret"], intent["lease_seconds"],
_later(intent["lease_seconds"]), _now(), intent["callback_ip"]),
)
else:
db.execute("DELETE FROM websub_subscriptions WHERE topic = ? AND callback = ?",
(intent["topic"], intent["callback"]))
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
(intent["id"], intent["request_id"]))
logger.info(f"WebSub {intent['mode']} verified: {intent['callback']} -> {intent['topic']}")
elif retry and intent["attempts"] < len(INTENT_BACKOFF):
db.execute(
"UPDATE websub_intents SET attempts = attempts + 1, next_attempt_at = ? "
"WHERE id = ? AND request_id IS ?",
(_later(INTENT_BACKOFF[intent["attempts"]]), intent["id"], intent["request_id"]),
)
else:
# Verification failed: the subscription state stays as it was
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
(intent["id"], intent["request_id"]))
logger.info(f"WebSub {intent['mode']} not verified for {intent['callback']}: {error}")
db.commit()
# --- content distribution (spec 7) ---------------------------------------
def render_topic(self, topic: str):
"""(status, content_type, body) of a topic, rendered in-app"""
path = urlparse(topic).path or "/"
response = self.app.test_client().get(path, headers={"Accept": "*/*"})
return response.status_code, response.headers.get("Content-Type", ""), response.get_data()
def deliver(self, db, delivery, rendered: Optional[dict] = None) -> None:
from starpunk.websub import hub_url
from starpunk.webmention_ssrf import ResolutionError, SSRFBlockedError, safe_post, user_agent
rendered = {} if rendered is None else rendered
site = self.app.config["SITE_URL"]
sub = None
if delivery["subscription_id"] is None:
# External hub: ping it (hub.url is the de facto parameter)
target, kwargs = hub_url(), {
"data": {"hub.mode": "publish", "hub.url": delivery["topic"], "hub.topic": delivery["topic"]},
"headers": {"User-Agent": user_agent(site)},
}
else:
sub = db.execute(
"SELECT * FROM websub_subscriptions WHERE id = ? AND expires_at > ?",
(delivery["subscription_id"], _now()),
).fetchone()
if sub is None: # unsubscribed or expired meanwhile
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
db.commit()
return
if sub["topic"] not in rendered:
rendered[sub["topic"]] = self.render_topic(sub["topic"])
status, content_type, body = rendered[sub["topic"]]
if status in (404, 410):
# The topic is gone: end the subscription and say so (spec 5.2)
self._deny(sub, "The topic no longer exists")
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
db.commit()
return
if status != 200:
self._retry_or_drop(db, delivery, f"Topic rendered HTTP {status}")
db.commit()
return
headers = {
"Content-Type": content_type,
"Link": f'<{hub_url()}>; rel="hub", <{sub["topic"]}>; rel="self"',
"User-Agent": user_agent(site),
}
if sub["secret"] is not None:
headers["X-Hub-Signature"] = signature(sub["secret"], body)
target, kwargs = sub["callback"], {"content": body, "headers": headers}
permanent = False
try:
result = safe_post(target, **kwargs)
status, error = result.status_code, f"HTTP {result.status_code}"
except ResolutionError as e:
status, error = None, str(e) # DNS can recover: retry
except (SSRFBlockedError, ValueError) as e:
status, error, permanent = None, f"Blocked: {e}", True
except httpx.HTTPError as e:
status, error = None, (str(e) or type(e).__name__)[:200]
succeeded = status is not None and 200 <= status < 300
if succeeded:
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
if sub is not None:
db.execute("UPDATE websub_subscriptions SET last_success_at = ?, last_error = NULL WHERE id = ?",
(_now(), sub["id"]))
elif sub is not None and status == 410:
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
logger.info(f"WebSub subscriber gone (410), removed: {sub['callback']}")
else:
if permanent:
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
logger.warning(f"WebSub delivery of {delivery['topic']} refused: {error}")
else:
self._retry_or_drop(db, delivery, error)
if sub is not None:
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
db.commit()
def _retry_or_drop(self, db, delivery, error: str) -> None:
"""
Retry a failed delivery with backoff, then give up on it
The subscription itself stays active until its lease ends (spec 7);
the next change is delivered again.
"""
if delivery["attempts"] < len(DELIVERY_BACKOFF):
db.execute(
"UPDATE websub_deliveries SET attempts = attempts + 1, next_attempt_at = ?, last_error = ? WHERE id = ?",
(_later(DELIVERY_BACKOFF[delivery["attempts"]]), error, delivery["id"]),
)
else:
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
logger.warning(f"WebSub delivery of {delivery['topic']} gave up: {error}")
def _deny(self, sub, reason: str) -> None:
"""Tell a subscriber its subscription has ended (hub.mode=denied), best effort"""
from starpunk.webmention_ssrf import safe_get, user_agent
url = callback_with_params(sub["callback"], {"hub.mode": "denied", "hub.topic": sub["topic"],
"hub.reason": reason})
try:
safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
max_redirects=0, max_bytes=VERIFY_MAX_BYTES)
except Exception as e:
logger.info(f"WebSub denial to {sub['callback']} not delivered: {e}")
# --- housekeeping -------------------------------------------------------
def prune(self, db) -> int:
"""Remove subscriptions whose lease has run out, and their deliveries"""
cursor = db.execute("DELETE FROM websub_subscriptions WHERE expires_at <= ?", (_now(),))
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND "
"subscription_id NOT IN (SELECT id FROM websub_subscriptions)")
db.commit()
self._last_prune = datetime.utcnow()
if cursor.rowcount:
logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions")
return cursor.rowcount
def start_worker(app) -> Optional[WebSubWorker]:
if not app.config.get("WEBSUB_ENABLED", True) or app.config.get("TESTING", False):
return None
worker = WebSubWorker(app, poll_interval=app.config.get("WEBSUB_POLL_INTERVAL", 5))
worker.start()
return worker
+15
View File
@@ -20,6 +20,21 @@
{% endif %}
</div>
{# WebSub subscribers (v2.10.0) #}
{% if websub_stats %}
<p class="note-meta websub-stats">
{% if websub_stats.external %}
WebSub: using {{ websub_stats.external }}
{% elif websub_stats.by_topic %}
WebSub subscribers:
{% for topic, count in websub_stats.by_topic %}{{ topic|replace(config.SITE_URL.rstrip('/'), '') or '/' }} {{ count }}{% if not loop.last %}, {% endif %}{% endfor %}
{% if websub_stats.retrying %} · {{ websub_stats.retrying }} deliveries being retried{% endif %}
{% else %}
WebSub: no subscribers yet
{% endif %}
</p>
{% endif %}
{% if notes %}
<div class="table-scroll">
<table class="note-table">
+7
View File
@@ -14,6 +14,13 @@
<link rel="webmention" href="/webmention">
{% endif %}
{# WebSub hub and topic on feed pages (v2.10.0) #}
{% set websub = websub_links() %}
{% if websub %}
<link rel="hub" href="{{ websub.hub }}">
<link rel="self" href="{{ websub.self }}">
{% endif %}
{# Micropub and IndieAuth discovery (independent of Webmention, v2.8.0) #}
<link rel="micropub" href="{{ absolute_url_for('micropub.micropub_endpoint') }}">
<link rel="indieauth-metadata" href="{{ absolute_url_for('indieauth.metadata') }}">
+1 -1
View File
@@ -230,7 +230,7 @@ class TestGenerateFeed:
# Check for Atom namespace and self-link
assert "atom" in feed_xml
assert "feed.xml" in feed_xml
assert "feed.rss" in feed_xml # self link (v2.10.0: /feed.rss is the RSS topic)
assert 'rel="self"' in feed_xml
def test_generate_feed_item_structure(self, app, sample_notes):
+318
View File
@@ -0,0 +1,318 @@
"""
WebSub publisher and built-in hub (v2.10.0, ADR-071)
Section numbers refer to the W3C WebSub Recommendation. Subscribers are
simulated with an httpx MockTransport behind the SSRF-safe fetcher.
"""
import hashlib
import hmac
import json
import sqlite3
from urllib.parse import parse_qs, urlparse
import httpx
import mf2py
import pytest
from starpunk.websub_worker import WebSubWorker
SITE = "http://localhost:5000"
HUB = f"{SITE}/websub"
PUBLIC = [(2, 1, 6, "", ("93.184.216.34", 0))]
@pytest.fixture
def web(monkeypatch):
"""Subscriber callbacks: handler(request) -> response, by host"""
hosts, seen = {}, []
# Every test client shares one IP; don't let earlier tests rate-limit this one
from starpunk.rate_limiter import RateLimiter
monkeypatch.setattr("starpunk.routes.websub._limiter", RateLimiter(limit=1000, window_seconds=60))
def handler(request):
host = request.headers["host"]
seen.append(request)
return hosts[host](request)
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", lambda *a, **k: PUBLIC)
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
return hosts, seen
def echo(request):
"""A subscriber that confirms every verification and accepts content"""
if request.method == "GET":
return httpx.Response(200, text=parse_qs(urlparse(str(request.url)).query)["hub.challenge"][0])
return httpx.Response(202)
def db(app):
conn = sqlite3.connect(app.config["DATABASE_PATH"])
conn.row_factory = sqlite3.Row
return conn
def run_worker(app, times=1):
worker = WebSubWorker(app)
with app.app_context():
for _ in range(times):
worker.process_once()
return worker
def subscribe(client, topic, callback="https://sub.example/cb?id=1", **extra):
form = {"hub.mode": "subscribe", "hub.topic": topic, "hub.callback": callback, **extra}
return client.post("/websub", data=form)
def make_note(app, content="Hello", **kw):
from starpunk.notes import create_note
with app.test_request_context():
return create_note(content, published=kw.pop("published", True), **kw)
def links(response):
return response.headers.getlist("Link")
# 4 Discovery -------------------------------------------------------------------
@pytest.mark.parametrize("path,self_url", [
("/", f"{SITE}/"),
("/feed.atom", f"{SITE}/feed.atom"),
("/feed.rss", f"{SITE}/feed.rss"),
("/feed.xml", f"{SITE}/feed.rss"),
("/feed.json", f"{SITE}/feed.json"),
("/likes", f"{SITE}/likes"),
])
def test_topics_advertise_hub_and_self(client, path, self_url):
response = client.get(path)
values = links(response)
assert f'<{HUB}>; rel="hub"' in values
assert [v for v in values if 'rel="self"' in v] == [f'<{self_url}>; rel="self"']
def test_negotiated_feed_self_is_the_format_served(client):
response = client.get("/feed", headers={"Accept": "application/atom+xml"})
assert f'<{SITE}/feed.atom>; rel="self"' in links(response)
def test_tag_page_is_a_topic(app, client):
make_note(app, tags=["Python"])
assert f'<{SITE}/tag/python>; rel="self"' in links(client.get("/tag/Python"))
def test_note_page_is_not_a_topic(app, client):
note = make_note(app)
assert not [v for v in links(client.get(f"/note/{note.slug}")) if 'rel="hub"' in v]
def test_embedded_links(client):
rels = mf2py.parse(doc=client.get("/").data.decode(), url=SITE + "/")["rels"]
assert rels["hub"] == [HUB] and rels["self"] == [f"{SITE}/"]
assert f'<link rel="hub" href="{HUB}"/>' in client.get("/feed.atom").data.decode()
assert f'<atom:link href="{HUB}" rel="hub"/>' in client.get("/feed.rss").data.decode()
assert json.loads(client.get("/feed.json").data)["hubs"] == [{"type": "WebSub", "url": HUB}]
def test_external_hub(app, client, web):
app.config["WEBSUB_HUB_URL"] = "https://hub.example/"
hosts, seen = web
hosts["hub.example"] = lambda r: httpx.Response(204)
assert '<https://hub.example/>; rel="hub"' in links(client.get("/"))
assert subscribe(client, f"{SITE}/").status_code == 400 # our hub is off
make_note(app)
run_worker(app)
pings = {parse_qs(r.content.decode())["hub.url"][0] for r in seen}
assert pings == {f"{SITE}/", f"{SITE}/feed.atom", f"{SITE}/feed.rss", f"{SITE}/feed.json"}
# 5.1 Subscription requests ----------------------------------------------------------
def test_subscription_request_is_accepted(app, client, web):
response = subscribe(client, f"{SITE}/feed.atom", **{"hub.secret": "s3cret", "extra": "ignored"})
assert response.status_code == 202
row = db(app).execute("SELECT * FROM websub_intents").fetchone()
assert (row["mode"], row["topic"], row["secret"]) == ("subscribe", f"{SITE}/feed.atom", "s3cret")
@pytest.mark.parametrize("form,reason", [
({"hub.mode": "watch"}, "hub.mode"),
({"hub.topic": "https://elsewhere.example/feed"}, "hub.topic"),
({"hub.topic": f"{SITE}/note/x"}, "hub.topic"),
({"hub.topic": f"{SITE}/tag/missing"}, "hub.topic"),
({"hub.callback": "ftp://sub.example/cb"}, "hub.callback"),
({"hub.secret": "x" * 200}, "hub.secret"),
])
def test_bad_requests(client, web, form, reason):
base = {"hub.mode": "subscribe", "hub.topic": f"{SITE}/", "hub.callback": "https://sub.example/cb"}
response = client.post("/websub", data={**base, **form})
assert response.status_code == 400 and reason in response.data.decode()
def test_private_callback_refused(client, monkeypatch):
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
lambda *a, **k: [(2, 1, 6, "", ("10.0.0.2", 0))])
assert subscribe(client, f"{SITE}/").status_code == 400
def test_per_host_limit(app, client, web):
app.config["WEBSUB_MAX_PER_HOST"] = 1
hosts, _ = web
hosts["sub.example"] = echo
subscribe(client, f"{SITE}/")
run_worker(app)
assert subscribe(client, f"{SITE}/feed.atom").status_code == 429
# 5.3 Verification of intent --------------------------------------------------------
def test_verification_activates_subscription(app, client, web):
hosts, seen = web
hosts["sub.example"] = echo
subscribe(client, f"{SITE}/", **{"hub.lease_seconds": "86400"})
run_worker(app)
query = parse_qs(urlparse(str(seen[0].url)).query)
assert query["id"] == ["1"] # the callback's own query string is kept
assert query["hub.mode"] == ["subscribe"] and query["hub.topic"] == [f"{SITE}/"]
assert query["hub.lease_seconds"] == ["86400"]
sub = db(app).execute("SELECT * FROM websub_subscriptions").fetchone()
assert sub["topic"] == f"{SITE}/" and sub["verified_at"]
assert db(app).execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0] == 0
@pytest.mark.parametrize("response", [
httpx.Response(200, text="wrong"),
httpx.Response(404),
httpx.Response(302, headers={"Location": "https://sub.example/other"}),
])
def test_failed_verification_changes_nothing(app, client, web, response):
hosts, _ = web
hosts["sub.example"] = lambda r: response
subscribe(client, f"{SITE}/")
run_worker(app)
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
assert db(app).execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0] == 0
def test_unsubscribe(app, client, web):
hosts, _ = web
hosts["sub.example"] = echo
subscribe(client, f"{SITE}/")
run_worker(app)
client.post("/websub", data={"hub.mode": "unsubscribe", "hub.topic": f"{SITE}/",
"hub.callback": "https://sub.example/cb?id=1"})
run_worker(app)
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
def test_resubscribe_changes_secret_only_once_verified(app, client, web):
hosts, _ = web
hosts["sub.example"] = echo
subscribe(client, f"{SITE}/", **{"hub.secret": "one"})
run_worker(app)
hosts["sub.example"] = lambda r: httpx.Response(404)
subscribe(client, f"{SITE}/", **{"hub.secret": "two"})
run_worker(app)
assert db(app).execute("SELECT secret FROM websub_subscriptions").fetchone()[0] == "one"
# 6, 7 Content distribution ---------------------------------------------------------
def _subscribed(app, client, web, topic=f"{SITE}/feed.atom", secret=None):
hosts, seen = web
hosts["sub.example"] = echo
extra = {"hub.secret": secret} if secret else {}
subscribe(client, topic, **extra)
run_worker(app)
seen.clear()
return hosts, seen
def test_new_note_is_delivered_signed(app, client, web):
hosts, seen = _subscribed(app, client, web, secret="s3cret")
make_note(app, "Fresh post")
run_worker(app)
(post,) = [r for r in seen if r.method == "POST"]
assert post.headers["content-type"].startswith("application/atom+xml")
assert b"Fresh post" in post.content
assert f'<{HUB}>; rel="hub"' in post.headers["link"] and f'<{SITE}/feed.atom>; rel="self"' in post.headers["link"]
expected = "sha256=" + hmac.new(b"s3cret", post.content, hashlib.sha256).hexdigest()
assert post.headers["x-hub-signature"] == expected
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
def test_no_signature_without_secret(app, client, web):
hosts, seen = _subscribed(app, client, web)
make_note(app)
run_worker(app)
assert "x-hub-signature" not in [r for r in seen if r.method == "POST"][0].headers
def test_drafts_are_not_delivered(app, client, web):
_subscribed(app, client, web)
make_note(app, published=False)
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
def test_tag_topics_include_removed_tags(app, client, web):
from starpunk.notes import update_note
note = make_note(app, tags=["old"])
_subscribed(app, client, web, topic=f"{SITE}/tag/old")
with app.test_request_context():
update_note(id=note.id, content="Hello", tags=["new"])
assert db(app).execute("SELECT topic FROM websub_deliveries").fetchone()[0] == f"{SITE}/tag/old"
def test_delete_is_delivered(app, client, web):
from starpunk.notes import delete_note
note = make_note(app, "Going away")
hosts, seen = _subscribed(app, client, web)
with app.test_request_context():
delete_note(id=note.id, soft=False)
run_worker(app)
(post,) = [r for r in seen if r.method == "POST"]
assert b"Going away" not in post.content
def test_410_removes_subscription(app, client, web):
hosts, _ = _subscribed(app, client, web)
hosts["sub.example"] = lambda r: httpx.Response(410)
make_note(app)
run_worker(app)
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
def test_failures_are_retried(app, client, web):
hosts, _ = _subscribed(app, client, web)
hosts["sub.example"] = lambda r: httpx.Response(500)
make_note(app)
run_worker(app)
row = db(app).execute("SELECT attempts, next_attempt_at, last_error FROM websub_deliveries").fetchone()
assert row["attempts"] == 1 and row["last_error"] == "HTTP 500"
def test_expired_subscriptions_get_nothing(app, client, web):
_subscribed(app, client, web)
conn = db(app)
conn.execute("UPDATE websub_subscriptions SET expires_at = datetime('now', '-1 minute')")
conn.commit()
make_note(app)
assert conn.execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
run_worker(app) # prunes
assert conn.execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
# Feeds are fresh right after a change (ADR-071 3) ------------------------------------
def test_feed_shows_new_note_immediately(app, client):
from starpunk.routes import public
make_note(app, "First")
assert b"First" in client.get("/feed.atom").data
make_note(app, "Second") # cached note list is stale now
assert b"Second" in client.get("/feed.atom").data
assert public._feed_cache["notes"] is not None
+316
View File
@@ -0,0 +1,316 @@
"""
WebSub hardening (v2.10.1): one test per finding of the v2.10.0 review
"""
import io
import socket
import sqlite3
import threading
import time
from unittest.mock import patch
from urllib.parse import parse_qs, urlparse
import httpx
import pytest
from PIL import Image
from tests.test_websub import ( # noqa: F401
HUB, PUBLIC, SITE, db, echo, make_note, run_worker, subscribe, web,
)
from starpunk.websub_worker import WebSubWorker
def jpeg(color="red"):
buf = io.BytesIO()
Image.new("RGB", (40, 30), color).save(buf, format="JPEG")
return buf.getvalue()
def intents(app):
return db(app).execute("SELECT * FROM websub_intents ORDER BY id").fetchall()
def subs(app):
return db(app).execute("SELECT * FROM websub_subscriptions ORDER BY id").fetchall()
def subscribed(app, client, web, topic=f"{SITE}/feed.atom", callback="https://sub.example/cb"):
hosts, seen = web
hosts.setdefault(urlparse(callback).hostname, echo)
subscribe(client, topic, callback=callback)
run_worker(app)
seen.clear()
return hosts, seen
# H1, H2: one bad item can't stop the worker -----------------------------------
def test_oversized_challenge_reply_does_not_wedge_worker(app, client, web):
hosts, _ = web
hosts["big.example"] = lambda r: httpx.Response(200, content=b"x" * 100_000)
hosts["good.example"] = echo
subscribe(client, f"{SITE}/", callback="https://big.example/cb")
subscribe(client, f"{SITE}/", callback="https://good.example/cb")
run_worker(app)
assert [s["callback"] for s in subs(app)] == ["https://good.example/cb"]
assert intents(app) == []
@pytest.mark.parametrize("callback", [
"https://example.com/\x00", "https://example.com/a b", "https://example.com:25/cb",
f"{SITE}/cb", "https://" + "a" * 2100 + ".example/",
])
def test_bad_callbacks_refused(client, web, callback):
assert subscribe(client, f"{SITE}/", callback=callback).status_code == 400
def test_unexpected_error_affects_only_that_item(app, client, web):
hosts, _ = web
hosts["good.example"] = echo
def crash(request):
raise RuntimeError("something nobody expected")
hosts["bad.example"] = crash
subscribe(client, f"{SITE}/", callback="https://bad.example/cb") # first in the queue
subscribe(client, f"{SITE}/", callback="https://good.example/cb")
run_worker(app)
assert [s["callback"] for s in subs(app)] == ["https://good.example/cb"]
assert intents(app) == []
def test_idn_callback_is_punycoded(app, client, web):
assert subscribe(client, f"{SITE}/", callback="https://bücher.example/cb").status_code == 202
assert intents(app)[0]["callback"] == "https://xn--bcher-kva.example/cb"
# H3, H4: photos are in what's delivered and in the feeds -----------------------
def test_new_photo_note_is_delivered_with_its_photo(app, client, web):
from starpunk.media import attach_media_to_note, save_media
hosts, seen = subscribed(app, client, web)
with app.test_request_context():
note = make_note(app, "With a photo", has_media=True)
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
media = save_media(jpeg(), "cat.jpg")
attach_media_to_note(note.id, [media["id"]], ["A cat"])
run_worker(app)
(post,) = [r for r in seen if r.method == "POST"]
assert media["path"].encode() in post.content
def test_photo_change_refreshes_feed_and_is_delivered(app, client, web):
from starpunk.media import attach_media_to_note, save_media
with app.test_request_context():
note = make_note(app, "Photo later")
first = save_media(jpeg("red"), "a.jpg")
attach_media_to_note(note.id, [first["id"]], [""])
assert first["path"].encode() in client.get("/feed.atom").data # cached now
hosts, seen = subscribed(app, client, web)
with app.test_request_context():
second = save_media(jpeg("blue"), "b.jpg")
attach_media_to_note(note.id, [second["id"]], [""])
feed = client.get("/feed.atom").data
assert second["path"].encode() in feed and first["path"].encode() not in feed
run_worker(app)
assert second["path"].encode() in [r for r in seen if r.method == "POST"][0].content
# H5: a slow subscriber can't hold the worker ------------------------------------
def test_slow_callback_is_cut_off():
from starpunk import webmention_ssrf as w
server = socket.socket()
server.bind(("127.0.0.1", 0))
server.listen(1)
port = server.getsockname()[1]
def trickle():
conn, _ = server.accept()
conn.recv(65536)
try:
for b in b"HTTP/1.1 200 OK\r\nX-Slow: " + b"y" * 100:
conn.send(bytes([b]))
time.sleep(0.3)
except OSError:
pass
finally:
conn.close()
threading.Thread(target=trickle, daemon=True).start()
start = time.monotonic()
with patch.object(w, "safe_resolve", lambda host: "127.0.0.1"), pytest.raises(httpx.TimeoutException):
w.safe_post(f"http://slow.example:{port}/", data={"a": "b"}, timeout=1)
assert time.monotonic() - start < 3
# Spec 7: subscriptions stay until their lease ends --------------------------------
def test_failing_subscriber_is_kept_until_lease_end(app, client, web):
hosts, _ = subscribed(app, client, web)
hosts["sub.example"] = lambda r: httpx.Response(500)
conn = db(app)
conn.execute("UPDATE websub_subscriptions SET verified_at = datetime('now', '-10 days')")
conn.commit()
make_note(app)
conn.execute("UPDATE websub_deliveries SET attempts = 5") # last retry
conn.commit()
run_worker(app)
assert conn.execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
assert len(subs(app)) == 1
def test_dns_failure_is_retried(app, client, web, monkeypatch):
subscribed(app, client, web)
make_note(app)
def fail(*a, **k):
raise socket.gaierror("temporary failure")
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", fail)
run_worker(app)
row = db(app).execute("SELECT attempts, last_error FROM websub_deliveries").fetchone()
assert row["attempts"] == 1 and "resolve" in row["last_error"]
def test_render_error_is_retried(app, client, web):
subscribed(app, client, web)
make_note(app)
with patch.object(WebSubWorker, "render_topic", lambda self, t: (500, "text/plain", b"")):
run_worker(app)
assert db(app).execute("SELECT attempts FROM websub_deliveries").fetchone()[0] == 1
def test_gone_topic_ends_subscription_with_denial(app, client, web):
hosts, seen = subscribed(app, client, web)
make_note(app)
with patch.object(WebSubWorker, "render_topic", lambda self, t: (404, "text/html", b"")):
run_worker(app)
assert subs(app) == []
denial = [r for r in seen if r.method == "GET"][0]
assert parse_qs(urlparse(str(denial.url)).query)["hub.mode"] == ["denied"]
# M1: limits count pending requests and the callback's address ----------------------
def test_limits_count_pending_requests(app, client, web):
app.config["WEBSUB_MAX_PER_HOST"] = 3
codes = [subscribe(client, f"{SITE}/", callback=f"https://s{i}.example/cb").status_code for i in range(5)]
assert codes == [202, 202, 202, 429, 429] # all resolve to one address
def test_renewal_is_allowed_at_the_limit(app, client, web):
app.config["WEBSUB_MAX_PER_HOST"] = 1
subscribed(app, client, web)
assert subscribe(client, f"{SITE}/feed.atom", callback="https://sub.example/cb").status_code == 202
# Spec 5.1 details -----------------------------------------------------------------
def test_unsubscribe_ignores_lease_and_secret(app, client, web):
form = {"hub.mode": "unsubscribe", "hub.topic": f"{SITE}/", "hub.callback": "https://sub.example/cb",
"hub.lease_seconds": "abc", "hub.secret": ""}
assert client.post("/websub", data=form).status_code == 202
def test_unreserved_characters_are_decoded(app, client, web):
assert subscribe(client, f"{SITE}/%66eed%2Eatom").status_code == 202
assert intents(app)[0]["topic"] == f"{SITE}/feed.atom"
def test_verification_echoes_the_requested_topic(app, client, web):
hosts, seen = web
hosts["sub.example"] = echo
make_note(app, tags=["Python"])
subscribe(client, f"{SITE}/tag/Python")
run_worker(app)
assert parse_qs(urlparse(str(seen[0].url)).query)["hub.topic"] == [f"{SITE}/tag/Python"]
assert subs(app)[0]["topic"] == f"{SITE}/tag/python"
def test_content_type_is_case_insensitive(client, web):
response = client.post("/websub", data="hub.mode=subscribe&hub.topic=" + SITE + "/&hub.callback=https://sub.example/cb",
content_type="Application/X-WWW-Form-Urlencoded")
assert response.status_code == 202
def test_empty_secret_is_refused(client, web):
assert subscribe(client, f"{SITE}/", **{"hub.secret": ""}).status_code == 400
def test_challenge_must_match(app, client, web):
hosts, _ = web
def padded(request):
challenge = parse_qs(urlparse(str(request.url)).query)["hub.challenge"][0]
return httpx.Response(200, text=f" {challenge} ")
hosts["sub.example"] = padded
subscribe(client, f"{SITE}/")
run_worker(app)
assert subs(app) == []
# L1: a newer request isn't lost while an older one is verified ------------------------
def test_newer_request_survives_older_verification(app, client, web):
hosts, _ = web
def slow_echo(request):
# A newer request (new secret) arrives while this one is being verified
subscribe(client, f"{SITE}/", **{"hub.secret": "new"})
return echo(request)
hosts["sub.example"] = slow_echo
subscribe(client, f"{SITE}/", **{"hub.secret": "old"})
run_worker(app)
pending = intents(app)
assert len(pending) == 1 and pending[0]["secret"] == "new"
# L2: fewer renders and pushes ------------------------------------------------------
def test_topic_rendered_once_per_pass(app, client, web):
hosts, _ = web
hosts["a.example"] = hosts["b.example"] = echo
for cb in ("https://a.example/cb", "https://b.example/cb"):
subscribe(client, f"{SITE}/feed.atom", callback=cb)
run_worker(app)
make_note(app)
calls = []
original = WebSubWorker.render_topic
def counting(self, topic):
calls.append(topic)
return original(self, topic)
with patch.object(WebSubWorker, "render_topic", counting):
run_worker(app)
assert calls == [f"{SITE}/feed.atom"]
def test_like_only_updates_likes_page(app, client, web):
hosts, _ = web
hosts["sub.example"] = echo
for topic in (f"{SITE}/", f"{SITE}/likes"):
subscribe(client, topic)
run_worker(app)
make_note(app, "", like_of="https://elsewhere.example/post")
topics = [r[0] for r in db(app).execute("SELECT topic FROM websub_deliveries")]
assert topics == [f"{SITE}/likes"]
# L5: feeds ----------------------------------------------------------------------
def test_etag_is_stable_across_processes(app, client):
from starpunk.feeds import get_cache
make_note(app)
first = client.get("/feed.rss").headers["ETag"]
get_cache().invalidate() # as if another process rendered it
time.sleep(1.1) # a different build time
assert client.get("/feed.rss").headers["ETag"] == first
def test_negotiated_feed_varies_on_accept(client):
assert "Accept" in client.get("/feed", headers={"Accept": "application/atom+xml"}).headers.get("Vary", "")