Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fa551de197 | ||
|
|
b2043286fa | ||
|
|
409b8b31b4 | ||
|
|
b520b8b4d3 | ||
|
|
720a1e6d6f |
No files matched your search
@@ -125,6 +125,24 @@ WORKER_TIMEOUT=30
|
||||
# Max requests per worker before restart (prevents memory leaks)
|
||||
MAX_REQUESTS=1000
|
||||
|
||||
# Behind a reverse proxy (Caddy, nginx): take the client address from
|
||||
# X-Forwarded-For. Without it every visitor shares the proxy's address, so one
|
||||
# busy client can rate-limit everyone on /webmention and /websub.
|
||||
# Only enable when the proxy sets X-Forwarded-For itself (Caddy does).
|
||||
TRUSTED_PROXY=true
|
||||
|
||||
# =============================================================================
|
||||
# WEBSUB (v2.10.0)
|
||||
# =============================================================================
|
||||
|
||||
# Built-in hub at /websub for this site's feeds. Set WEBSUB_HUB_URL to use
|
||||
# another hub instead (StarPunk then pings it on changes).
|
||||
WEBSUB_ENABLED=true
|
||||
# WEBSUB_HUB_URL=https://hub.example.com/
|
||||
# WEBSUB_MAX_SUBSCRIPTIONS=1000
|
||||
# Per callback server (resolved address)
|
||||
# WEBSUB_MAX_PER_HOST=25
|
||||
|
||||
# =============================================================================
|
||||
# DEVELOPMENT OPTIONS
|
||||
# =============================================================================
|
||||
|
||||
@@ -7,6 +7,49 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [2.10.1] - 2026-11-03
|
||||
|
||||
WebSub fixes from an adversarial review of v2.10.0 against the [W3C Recommendation](https://www.w3.org/TR/websub/) (details in ADR-071, Amendments).
|
||||
|
||||
### Security
|
||||
|
||||
- One subscription request could stop the hub for good: a callback that answered verification with a large page, or a callback URL with a control character, crashed the worker on every pass. Each queued item is now handled on its own
|
||||
- A subscriber sending its response one byte at a time could hold the hub for hours. Every request the hub (and webmention fetching) makes now has a hard time limit
|
||||
- The subscription limits only counted verified subscriptions and could be dodged with subdomains. They now count pending requests too, per callback server address, and at most 200 requests can wait for verification
|
||||
- Callbacks must be printable URLs of reasonable length, not on this site, and on ports 80, 443 or 1024 and up
|
||||
|
||||
### Fixed
|
||||
|
||||
- New posts with photos were sent to subscribers (and cached in the feeds) without their photos, and photo-only edits sent the old feed. Notes with photos are announced once the photos are attached, and the feed cache follows photo and caption changes
|
||||
- A subscriber whose deliveries failed for three days was removed; WebSub says to keep it until its lease ends. Deliveries are retried and given up, the subscription stays
|
||||
- A temporary DNS failure or an error rendering the feed lost the delivery instead of retrying it. A feed that no longer exists now ends its subscriptions with a `hub.mode=denied` notice
|
||||
- Verification sent our normalised topic (`/tag/python`) instead of the one requested (`/tag/Python`), so strict subscribers refused it. Percent-encoded topics (`/%66eed.atom`) are now understood, the Content-Type check ignores case, unsubscribe requests ignore `hub.lease_seconds`, and an empty `hub.secret` is refused instead of turning signatures off
|
||||
- A newer subscription request arriving while an older one was being verified was lost (migration 017)
|
||||
- Unsubscribing works while `WEBSUB_HUB_URL` points to another hub
|
||||
- Feed ETags differed between server processes, so conditional requests rarely matched; `/feed` now sends `Vary: Accept`
|
||||
- A like no longer causes pushes of the home page and feeds, which don't show likes; each feed is rendered once per delivery round, not once per subscriber
|
||||
- Internationalised callback domains can be verified
|
||||
|
||||
### Changed
|
||||
|
||||
- `.env.example` documents `TRUSTED_PROXY` (needed behind a reverse proxy for per-client rate limits) and the WebSub settings
|
||||
|
||||
## [2.10.0] - 2026-11-03
|
||||
|
||||
### Added
|
||||
|
||||
- **WebSub** ([ADR-071](docs/decisions/ADR-071-websub.md)): readers that support it get new posts, edits and deletions within seconds instead of polling. StarPunk runs its own hub at `/websub` for its own feeds: the home page, `/feed.atom`, `/feed.rss`, `/feed.json`, `/likes` and each tag page. Each advertises the hub and itself (`Link` headers, and links inside the page or feed). Subscriptions are verified with a challenge, leases expire, deliveries carry the full feed, signed with `X-Hub-Signature` when the subscriber gave a secret, and failures are retried for about nine hours. Requests to subscribers go through the same private-address protections as webmentions, and there are limits per subscriber host and in total. Set `WEBSUB_HUB_URL` to use another hub instead; `WEBSUB_ENABLED=false` turns it off
|
||||
- The admin dashboard shows WebSub subscribers per feed
|
||||
|
||||
### Fixed
|
||||
|
||||
- Feeds could be up to five minutes out of date after posting, editing or deleting, and each server process cached its own copy. Any change to a public note now refreshes the feeds everywhere at once
|
||||
- The memory monitor stopped after the first request (the same bug the webmention worker had before v2.6.8)
|
||||
|
||||
### Changed
|
||||
|
||||
- The RSS feed calls itself `/feed.rss` instead of the older `/feed.xml`. Both addresses still work
|
||||
|
||||
## [2.9.0] - 2026-11-03
|
||||
|
||||
Micropub review against the [W3C Recommendation](https://www.w3.org/TR/micropub/).
|
||||
|
||||
@@ -0,0 +1,206 @@
|
||||
# ADR-071: WebSub publishing with a built-in hub
|
||||
|
||||
## Status
|
||||
|
||||
Accepted (v2.10.0, 2026-11-03)
|
||||
|
||||
## Context
|
||||
|
||||
Readers poll StarPunk's feeds. WebSub (W3C Recommendation,
|
||||
https://www.w3.org/TR/websub/) lets readers that support it (Monocle,
|
||||
Aperture, Inoreader, NewsBlur, Miniflux and others) subscribe once and
|
||||
have new content pushed to them within seconds.
|
||||
|
||||
WebSub has three roles. StarPunk is the **publisher**. Someone has to run
|
||||
the **hub** that keeps subscriptions and delivers content. Public hubs
|
||||
exist (switchboard.p3k.io, pubsubhubbub.appspot.com), but they are a
|
||||
third-party dependency and some have shut down. The rest of this site is
|
||||
self-hosted.
|
||||
|
||||
ADR-014 deferred WebSub for v1 as "complexity, external dependency".
|
||||
|
||||
Two existing problems matter here:
|
||||
|
||||
- Feeds are cached for `FEED_CACHE_SECONDS` (5 minutes) and nothing clears
|
||||
the cache when a note changes. A hub fetching right after a change would
|
||||
get stale content. Regular readers are also up to 5 minutes behind.
|
||||
- The RSS feed is served at `/feed.rss` and `/feed.xml` (legacy alias), and
|
||||
calls itself `/feed.xml`.
|
||||
|
||||
## Decision
|
||||
|
||||
StarPunk runs its own hub, **for its own topics only**, at `/websub`.
|
||||
`WEBSUB_HUB_URL` can point at an external hub instead; then StarPunk
|
||||
advertises that hub, pings it on changes, and its own `/websub` is off.
|
||||
|
||||
### 1. Topics
|
||||
|
||||
Absolute URLs under `SITE_URL`:
|
||||
|
||||
| Topic | Content type |
|
||||
|---|---|
|
||||
| `/` (h-feed) | text/html |
|
||||
| `/feed.atom` | application/atom+xml |
|
||||
| `/feed.rss` | application/rss+xml |
|
||||
| `/feed.json` | application/feed+json |
|
||||
| `/likes` | text/html |
|
||||
| `/tag/<name>` | text/html |
|
||||
|
||||
`/feed.xml` and the negotiated `/feed` advertise the topic of the format
|
||||
they serve as `rel=self` (`/feed.rss`, `/feed.atom`, `/feed.json`). The
|
||||
RSS feed's own self link changes from `/feed.xml` to `/feed.rss`.
|
||||
|
||||
### 2. Discovery (publisher, spec 4)
|
||||
|
||||
Every topic response carries `Link: <hub>; rel="hub"` and exactly one
|
||||
`Link: <topic>; rel="self"`. Embedded links as well: `<link rel="hub">` and
|
||||
`<link rel="self">` in the HTML head of topic pages, `<link rel="hub">`
|
||||
next to the existing self link in Atom, `<atom:link rel="hub">` in RSS, and
|
||||
`"hubs": [{"type": "WebSub", "url": ...}]` in JSON Feed.
|
||||
|
||||
### 3. What changed, and when
|
||||
|
||||
`websub.note_changed(note, old_tags=())` is called wherever
|
||||
`syndication.note_changed` is, plus on create. It:
|
||||
|
||||
1. clears the feed cache (all formats), and
|
||||
2. if the note is or was public, queues delivery for `/`, the three
|
||||
feeds, `/likes` if it is a like, and each `/tag/<name>` for its tags
|
||||
before and after the change.
|
||||
|
||||
Drafts that stay drafts change nothing.
|
||||
|
||||
### 4. Hub endpoint (spec 5)
|
||||
|
||||
`POST /websub`, `application/x-www-form-urlencoded`:
|
||||
|
||||
- `hub.mode` subscribe or unsubscribe; `hub.topic` must be one of the
|
||||
topics above (after normalising), else 400 with a plain-text reason;
|
||||
`hub.callback` must be an absolute http(s) URL that passes the SSRF
|
||||
check; `hub.secret` under 200 bytes; `hub.lease_seconds` clamped to
|
||||
1 day to 30 days (default 10 days). Unknown parameters are ignored.
|
||||
- Response: 202 Accepted, independent of what verification later finds.
|
||||
- Limits: the Webmention rate limiter (per IP); at most 25 subscriptions
|
||||
per callback host and 1,000 in total; beyond that, 429.
|
||||
|
||||
### 5. Verification of intent (spec 5.3)
|
||||
|
||||
A background worker sends `GET <callback>` with the callback's query
|
||||
string kept and `hub.mode`, `hub.topic`, `hub.challenge` (random, 32
|
||||
bytes) and `hub.lease_seconds` appended. Success is a 2xx with the body
|
||||
equal to the challenge; anything else, including redirects, fails and
|
||||
leaves the subscription state unchanged. On success a subscribe becomes
|
||||
active with `expires_at = now + lease`, and an unsubscribe deletes it.
|
||||
A re-subscribe replaces the secret and lease only once verified.
|
||||
|
||||
### 6. Content distribution (spec 6, 7)
|
||||
|
||||
For each change, one delivery per active, unexpired subscription to the
|
||||
topic. The worker renders the topic inside the app (no HTTP round trip
|
||||
through Caddy), so subscribers get exactly what the public URL serves,
|
||||
then `POST`s it to the callback with:
|
||||
|
||||
- `Content-Type` of the topic,
|
||||
- `Link` headers with `rel=hub` and `rel=self`,
|
||||
- `X-Hub-Signature: sha256=<HMAC-SHA256(secret, body)>` if a secret was
|
||||
given.
|
||||
|
||||
A 2xx is success. 410 Gone removes the subscription. Anything else is
|
||||
retried after 1 min, 5 min, 30 min, 2 h and 6 h, then dropped; a
|
||||
subscription whose deliveries fail for 3 days is removed. Only the latest
|
||||
content is sent: a queued delivery for a topic is replaced by a newer one.
|
||||
|
||||
### 7. Outbound safety
|
||||
|
||||
Callbacks are arbitrary URLs, so verification and delivery go through
|
||||
`webmention_ssrf` (`safe_get`/`safe_post`): public addresses only, the
|
||||
connection pinned to the checked address, no redirects, 10 s time limit,
|
||||
1 MB response cap.
|
||||
|
||||
### 8. Data (migration 016)
|
||||
|
||||
- `websub_subscriptions(id, topic, callback, secret, lease_seconds,
|
||||
state ['pending','active'], expires_at, created_at, verified_at,
|
||||
last_success_at, last_error)`, unique on `(topic, callback)`.
|
||||
- `websub_intents(id, topic, callback, mode, secret, lease_seconds,
|
||||
attempts, next_attempt_at, created_at)`: verifications to do.
|
||||
- `websub_deliveries(id, subscription_id, attempts, next_attempt_at,
|
||||
last_error, created_at)`, unique on `subscription_id`.
|
||||
|
||||
### 9. Worker
|
||||
|
||||
The same pattern as the Webmention worker: one thread per gunicorn
|
||||
process, a file lock so only one process works the queues, app context
|
||||
around each poll. Expired subscriptions are pruned once a day.
|
||||
|
||||
### 10. Admin
|
||||
|
||||
The admin dashboard shows active subscribers per topic and failing
|
||||
deliveries.
|
||||
|
||||
### 11. Configuration
|
||||
|
||||
`WEBSUB_ENABLED` (default true), `WEBSUB_HUB_URL` (default: built-in),
|
||||
`WEBSUB_MAX_SUBSCRIPTIONS` (1000), `WEBSUB_MAX_PER_HOST` (25).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Readers that support WebSub get posts and edits within seconds.
|
||||
- Feeds are fresh right after a change for every reader, because the cache
|
||||
is cleared.
|
||||
- StarPunk sends HTTP requests to subscriber-chosen URLs. The SSRF
|
||||
protections, per-host and total limits, and rate limiting bound that.
|
||||
- websub.rocks' hub tests use websub.rocks' own topics, which this hub
|
||||
refuses by design. Testing uses unit tests of each spec section and a
|
||||
real subscription from a reader after deploy.
|
||||
- The RSS self link changes to `/feed.rss`. Readers keep whichever URL
|
||||
they subscribed to; both keep working.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Public hub only.** Least code, but a third-party dependency for a
|
||||
self-hosted site. Still available through `WEBSUB_HUB_URL`.
|
||||
- **A separate hub service** (another container). More to deploy and
|
||||
keep updated, and few maintained hubs exist.
|
||||
- **Accepting any topic** (a general-purpose hub). Not needed, and it
|
||||
would let the hub be used to fetch and push arbitrary URLs.
|
||||
|
||||
## Amendments (v2.10.1, 2026-11-03)
|
||||
|
||||
An adversarial review of v2.10.0 against the Recommendation led to these
|
||||
changes; where they differ, they replace the text above.
|
||||
|
||||
- **Worker robustness.** Each queued item is handled on its own: an
|
||||
unexpected error drops (verification) or retries (delivery) that item
|
||||
only. Before, a callback answering with a large page or containing a
|
||||
control character stopped the worker for good.
|
||||
- **Time limits.** `safe_get` and `safe_post` close the connection when
|
||||
the whole exchange exceeds its time limit (10 s), not just a single read.
|
||||
Verification replies over 4 KB fail; delivery responses are read up to
|
||||
4 KB.
|
||||
- **Leases are kept (spec 7).** A failing subscriber is not removed; its
|
||||
deliveries are retried and then given up, and the next change is
|
||||
delivered again. Only lease expiry, 410 Gone or unsubscribing end a
|
||||
subscription. DNS failures are retried. A topic that renders 404/410
|
||||
ends its subscriptions with a `hub.mode=denied` notice; other render
|
||||
errors are retried.
|
||||
- **Photos.** A note with photos is announced once they are attached
|
||||
(`attach_media_to_note` notifies), and the feed cache key and ETag
|
||||
include `content_version`, so photo and caption changes reach feeds and
|
||||
subscribers. ETags no longer depend on the feed's build time.
|
||||
- **Subscription requests (spec 5.1).** Unreserved percent-encodings are
|
||||
decoded; the Content-Type check is case-insensitive; `hub.secret` and
|
||||
`hub.lease_seconds` are ignored for unsubscriptions; an empty secret is
|
||||
refused; verification echoes `hub.topic` exactly as requested
|
||||
(`requested_topic`); unsubscribing works while an external hub is
|
||||
configured. Callbacks must be printable, at most 2,048 characters, not on
|
||||
this site, on port 80, 443 or 1024+, and are stored with an IDNA host.
|
||||
- **Limits** count pending requests as well as subscriptions, per callback
|
||||
*address* (so subdomains on one server share a limit), with at most 200
|
||||
pending requests. Rate limiting needs `TRUSTED_PROXY=true` behind Caddy.
|
||||
- **Races.** Each request has a `request_id`; verifying an older request
|
||||
doesn't consume a newer one (migration 017).
|
||||
- **Less work.** A topic is rendered once per worker pass; likes only
|
||||
update `/likes` and tag pages.
|
||||
- Corrections: there is no `state` column (a subscription row exists only
|
||||
once verified); expired leases are pruned hourly.
|
||||
@@ -0,0 +1,55 @@
|
||||
-- Migration 016: WebSub hub (v2.10.0, ADR-071)
|
||||
|
||||
-- Active and pending subscriptions to this site's topics
|
||||
CREATE TABLE IF NOT EXISTS websub_subscriptions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
topic TEXT NOT NULL,
|
||||
callback TEXT NOT NULL,
|
||||
secret TEXT,
|
||||
lease_seconds INTEGER NOT NULL,
|
||||
expires_at TIMESTAMP NOT NULL,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
verified_at TIMESTAMP,
|
||||
last_success_at TIMESTAMP,
|
||||
last_error TEXT,
|
||||
UNIQUE(topic, callback)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_topic ON websub_subscriptions(topic);
|
||||
|
||||
-- Subscribe/unsubscribe requests waiting for verification of intent
|
||||
CREATE TABLE IF NOT EXISTS websub_intents (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
topic TEXT NOT NULL,
|
||||
callback TEXT NOT NULL,
|
||||
mode TEXT NOT NULL CHECK (mode IN ('subscribe', 'unsubscribe')),
|
||||
secret TEXT,
|
||||
lease_seconds INTEGER,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
UNIQUE(topic, callback)
|
||||
);
|
||||
|
||||
-- Content to push: one row per subscription (built-in hub) or per topic
|
||||
-- (ping to an external hub, subscription_id NULL)
|
||||
CREATE TABLE IF NOT EXISTS websub_deliveries (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
subscription_id INTEGER REFERENCES websub_subscriptions(id) ON DELETE CASCADE,
|
||||
topic TEXT NOT NULL,
|
||||
attempts INTEGER NOT NULL DEFAULT 0,
|
||||
next_attempt_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
last_error TEXT,
|
||||
created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_subscription
|
||||
ON websub_deliveries(subscription_id) WHERE subscription_id IS NOT NULL;
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS idx_websub_deliveries_ping
|
||||
ON websub_deliveries(topic) WHERE subscription_id IS NULL;
|
||||
|
||||
-- Site-wide values. content_version changes whenever a public note does, so
|
||||
-- every gunicorn process knows its cached feed note list is stale.
|
||||
CREATE TABLE IF NOT EXISTS site_state (
|
||||
key TEXT PRIMARY KEY,
|
||||
value TEXT NOT NULL
|
||||
);
|
||||
INSERT OR IGNORE INTO site_state (key, value) VALUES ('content_version', '0');
|
||||
@@ -0,0 +1,12 @@
|
||||
-- Migration 017: WebSub hardening (v2.10.1)
|
||||
-- request_id: each subscription request gets its own id, so verifying an
|
||||
-- older request can't consume a newer one for the same topic and callback.
|
||||
-- requested_topic: the hub.topic exactly as sent; verification echoes it.
|
||||
-- callback_ip: address the callback resolved to, for per-address limits.
|
||||
|
||||
ALTER TABLE websub_intents ADD COLUMN request_id TEXT;
|
||||
ALTER TABLE websub_intents ADD COLUMN requested_topic TEXT;
|
||||
ALTER TABLE websub_intents ADD COLUMN callback_ip TEXT;
|
||||
ALTER TABLE websub_subscriptions ADD COLUMN callback_ip TEXT;
|
||||
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_ip ON websub_subscriptions(callback_ip);
|
||||
CREATE INDEX IF NOT EXISTS idx_websub_intents_ip ON websub_intents(callback_ip);
|
||||
+15
-8
@@ -309,6 +309,11 @@ def create_app(config=None):
|
||||
response.headers.add('Link', '</webmention>; rel="webmention"')
|
||||
return response
|
||||
|
||||
# WebSub discovery: Link rel=hub and rel=self on topics (v2.10.0)
|
||||
from starpunk.websub import add_discovery_headers, template_links
|
||||
app.after_request(add_discovery_headers)
|
||||
app.add_template_global(template_links, 'websub_links')
|
||||
|
||||
# Register centralized error handlers
|
||||
from starpunk.errors import register_error_handlers
|
||||
|
||||
@@ -322,12 +327,8 @@ def create_app(config=None):
|
||||
memory_monitor.start()
|
||||
app.memory_monitor = memory_monitor
|
||||
app.logger.info(f"Memory monitor started (interval={memory_monitor.interval}s)")
|
||||
|
||||
# Register cleanup handler
|
||||
@app.teardown_appcontext
|
||||
def cleanup_memory_monitor(error=None):
|
||||
if hasattr(app, 'memory_monitor') and app.memory_monitor.is_alive():
|
||||
app.memory_monitor.stop()
|
||||
# No teardown_appcontext stop: that ran after every request and stopped
|
||||
# the monitor straight away (v2.10.0). It is a daemon thread.
|
||||
|
||||
# Start webmention worker thread (v1.6.0 Phase 2)
|
||||
# Per background-worker.md: Skip in test mode
|
||||
@@ -350,6 +351,12 @@ def create_app(config=None):
|
||||
f"batch_size={webmention_worker.batch_size})"
|
||||
)
|
||||
|
||||
# Start WebSub worker (v2.10.0)
|
||||
from starpunk.websub_worker import start_worker
|
||||
app.websub_worker = start_worker(app)
|
||||
if app.websub_worker:
|
||||
app.logger.info("WebSub worker started")
|
||||
|
||||
# Start syndication worker when a target is configured (v2.1.0)
|
||||
from starpunk.syndication import SyndicationWorker, targets_configured
|
||||
if targets_configured(app.config) and not app.config.get("TESTING", False):
|
||||
@@ -463,5 +470,5 @@ def create_app(config=None):
|
||||
|
||||
# Package version (Semantic Versioning 2.0.0)
|
||||
# See docs/standards/versioning-strategy.md for details
|
||||
__version__ = "2.9.0"
|
||||
__version_info__ = (2, 9, 0)
|
||||
__version__ = "2.10.1"
|
||||
__version_info__ = (2, 10, 1)
|
||||
@@ -111,6 +111,13 @@ def load_config(app, config_override=None):
|
||||
|
||||
# Webmention configuration (v1.6.0)
|
||||
app.config["WEBMENTION_ENABLED"] = os.getenv("WEBMENTION_ENABLED", "true").lower() == "true"
|
||||
|
||||
# WebSub (v2.10.0, ADR-071): built-in hub unless WEBSUB_HUB_URL names another
|
||||
app.config["WEBSUB_ENABLED"] = os.getenv("WEBSUB_ENABLED", "true").lower() == "true"
|
||||
app.config["WEBSUB_HUB_URL"] = os.getenv("WEBSUB_HUB_URL", "").strip()
|
||||
app.config["WEBSUB_MAX_SUBSCRIPTIONS"] = int(os.getenv("WEBSUB_MAX_SUBSCRIPTIONS", "1000"))
|
||||
app.config["WEBSUB_MAX_PER_HOST"] = int(os.getenv("WEBSUB_MAX_PER_HOST", "25"))
|
||||
app.config["WEBSUB_POLL_INTERVAL"] = int(os.getenv("WEBSUB_POLL_INTERVAL", "5"))
|
||||
app.config["WEBMENTION_WORKER_ENABLED"] = os.getenv("WEBMENTION_WORKER_ENABLED", "true").lower() == "true"
|
||||
app.config["WEBMENTION_POLL_INTERVAL"] = int(os.getenv("WEBMENTION_POLL_INTERVAL", "10"))
|
||||
app.config["WEBMENTION_ERROR_BACKOFF"] = int(os.getenv("WEBMENTION_ERROR_BACKOFF", "60"))
|
||||
|
||||
@@ -169,7 +169,11 @@ class FeedCache:
|
||||
- Adds new entry or updates existing entry
|
||||
"""
|
||||
cache_key = self._generate_cache_key(format_name, notes_checksum)
|
||||
etag = self._generate_etag(content)
|
||||
# From what the feed is made of, not its bytes: the bytes include a
|
||||
# build time, so each server process made a different ETag and
|
||||
# conditional requests rarely matched (v2.10.1). Weak ETag: same
|
||||
# notes, semantically the same feed.
|
||||
etag = self._generate_etag(f"{format_name}:{notes_checksum}")
|
||||
timestamp = time.time()
|
||||
|
||||
# Evict if cache is full
|
||||
|
||||
@@ -93,7 +93,7 @@ def generate_rss(
|
||||
fg.language("en")
|
||||
|
||||
# Add self-link for feed discovery (Atom namespace)
|
||||
fg.link(href=f"{site_url}/feed.xml", rel="self", type="application/rss+xml")
|
||||
fg.link(href=f"{site_url}/feed.rss", rel="self", type="application/rss+xml")
|
||||
|
||||
# Set last build date to now
|
||||
fg.lastBuildDate(datetime.now(timezone.utc))
|
||||
@@ -246,7 +246,7 @@ def generate_rss_streaming(
|
||||
yield f" <description>{_escape_xml(site_description or site_name)}</description>\n"
|
||||
yield " <language>en</language>\n"
|
||||
yield f" <lastBuildDate>{last_build}</lastBuildDate>\n"
|
||||
yield f' <atom:link href="{_escape_xml(site_url)}/feed.xml" rel="self" type="application/rss+xml"/>\n'
|
||||
yield f' <atom:link href="{_escape_xml(site_url)}/feed.rss" rel="self" type="application/rss+xml"/>\n'
|
||||
|
||||
# Yield items (newest first)
|
||||
# Notes from database are already in DESC order (newest first)
|
||||
|
||||
@@ -891,6 +891,15 @@ def attach_media_to_note(note_id: int, media_ids: List[int], captions: List[str]
|
||||
db.rollback()
|
||||
raise
|
||||
|
||||
# Feeds and pages show the photos: refresh caches and tell WebSub (v2.10.1)
|
||||
try:
|
||||
from starpunk.notes import _websub_changed, get_note
|
||||
note = get_note(id=note_id, load_content=False)
|
||||
if note is not None:
|
||||
_websub_changed(note, was_public=note.published)
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue WebSub update for note {note_id}: {e}")
|
||||
|
||||
|
||||
def get_note_media(note_id: int) -> List[Dict]:
|
||||
"""
|
||||
|
||||
@@ -760,7 +760,7 @@ def handle_update(data: dict, token_info: dict):
|
||||
if "photo" in touched:
|
||||
from starpunk.media import attach_media_to_note
|
||||
attach_media_to_note(note.id, media_ids, captions)
|
||||
note_changed(note.id)
|
||||
note_changed(note.id) # attach_media_to_note tells WebSub
|
||||
except InvalidNoteDataError as e:
|
||||
raise MicropubValidationError(str(e))
|
||||
except MicropubError:
|
||||
|
||||
+36
-1
@@ -374,9 +374,31 @@ def create_note(
|
||||
# Reply context fetch failure should not prevent note creation
|
||||
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
|
||||
|
||||
# A note with photos is announced when they're attached
|
||||
# (attach_media_to_note), so subscribers never get it without them
|
||||
if not has_media:
|
||||
_websub_changed(note)
|
||||
return note
|
||||
|
||||
|
||||
def _tag_names(note_id: int) -> list[str]:
|
||||
"""A note's tag names, for WebSub tag topics (best effort)"""
|
||||
try:
|
||||
from starpunk.tags import get_note_tags
|
||||
return [t["name"] for t in get_note_tags(note_id)]
|
||||
except Exception:
|
||||
return []
|
||||
|
||||
|
||||
def _websub_changed(note, was_public: bool = False, old_tags=()) -> None:
|
||||
"""Clear cached feeds and queue WebSub deliveries (v2.10.0, best effort)"""
|
||||
try:
|
||||
from starpunk import websub
|
||||
websub.note_changed(note, was_public=was_public, old_tags=old_tags)
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue WebSub update for note {note.slug}: {e}")
|
||||
|
||||
|
||||
def get_note(
|
||||
slug: Optional[str] = None, id: Optional[int] = None, load_content: bool = True
|
||||
) -> Optional[Note]:
|
||||
@@ -705,6 +727,7 @@ def update_note(
|
||||
if existing_note is None:
|
||||
identifier = slug if slug is not None else id
|
||||
raise NoteNotFoundError(identifier)
|
||||
old_tag_names = _tag_names(existing_note.id) # for WebSub tag topics (v2.10.0)
|
||||
|
||||
if empty_content and not (existing_note.reacts_to or has_media):
|
||||
raise InvalidNoteDataError(
|
||||
@@ -896,6 +919,8 @@ def update_note(
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue syndication update: {e}")
|
||||
|
||||
_websub_changed(updated_note, was_public=existing_note.published, old_tags=old_tag_names)
|
||||
|
||||
# 10. RETURN UPDATED NOTE
|
||||
return updated_note
|
||||
|
||||
@@ -995,6 +1020,10 @@ def delete_note(
|
||||
# For idempotency, don't raise error - just return
|
||||
return
|
||||
|
||||
# What feeds showed before, for WebSub (v2.10.0)
|
||||
was_public = bool(existing_note.published) and not getattr(existing_note, "deleted_at", None)
|
||||
deleted_tags = _tag_names(existing_note.id)
|
||||
|
||||
# 4. SETUP
|
||||
data_dir = Path(current_app.config["DATA_PATH"])
|
||||
note_path = data_dir / existing_note.file_path
|
||||
@@ -1081,6 +1110,10 @@ def delete_note(
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue syndication delete: {e}")
|
||||
|
||||
# Feeds and pages no longer show it (v2.10.0)
|
||||
object.__setattr__(existing_note, "published", False)
|
||||
_websub_changed(existing_note, was_public=was_public, old_tags=deleted_tags)
|
||||
|
||||
# 6. RETURN (no value)
|
||||
return None
|
||||
|
||||
@@ -1137,4 +1170,6 @@ def undelete_note(slug: str) -> Note:
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue webmentions after undelete: {e}")
|
||||
|
||||
return get_note(slug=slug)
|
||||
restored = get_note(slug=slug)
|
||||
_websub_changed(restored)
|
||||
return restored
|
||||
@@ -7,7 +7,7 @@ admin, auth, and (conditionally) dev auth routes.
|
||||
|
||||
from flask import Flask
|
||||
|
||||
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention
|
||||
from starpunk.routes import admin, auth, indieauth, micropub, public, search, webmention, websub
|
||||
|
||||
|
||||
def register_routes(app: Flask) -> None:
|
||||
@@ -44,6 +44,9 @@ def register_routes(app: Flask) -> None:
|
||||
if app.config.get("WEBMENTION_ENABLED", True):
|
||||
app.register_blueprint(webmention.bp)
|
||||
|
||||
# WebSub hub (v2.10.0); refuses requests when disabled or using another hub
|
||||
app.register_blueprint(websub.bp)
|
||||
|
||||
# Conditionally register dev auth routes
|
||||
if app.config.get("DEV_MODE"):
|
||||
app.logger.warning(
|
||||
|
||||
@@ -72,12 +72,23 @@ def dashboard():
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# WebSub subscribers (v2.10.0)
|
||||
websub_stats = None
|
||||
if current_app.config.get('WEBSUB_ENABLED', True):
|
||||
try:
|
||||
from starpunk import websub
|
||||
from starpunk.database import get_db
|
||||
websub_stats = websub.stats(get_db(current_app))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
return render_template(
|
||||
"admin/dashboard.html",
|
||||
notes=notes,
|
||||
page=page,
|
||||
total_pages=total_pages,
|
||||
webmention_pending_count=webmention_pending_count,
|
||||
websub_stats=websub_stats,
|
||||
user_me=g.me
|
||||
)
|
||||
|
||||
@@ -139,13 +150,17 @@ def create_note_submit():
|
||||
# Parse tags (v1.3.0 Phase 3)
|
||||
tags = parse_tag_input(tags_input)
|
||||
|
||||
# With photos, WebSub hears about the note once they're attached (v2.10.1)
|
||||
has_uploads = any(f.filename for f in request.files.getlist('media_files'))
|
||||
|
||||
try:
|
||||
# Create note first (per Q4)
|
||||
note = create_note(
|
||||
content,
|
||||
published=published,
|
||||
custom_slug=custom_slug if custom_slug else None,
|
||||
tags=tags if tags else None
|
||||
tags=tags if tags else None,
|
||||
has_media=has_uploads,
|
||||
)
|
||||
|
||||
# Handle media uploads (v1.2.0 Phase 3)
|
||||
@@ -184,6 +199,11 @@ def create_note_submit():
|
||||
if errors:
|
||||
flash(f"Note created, but some images failed: {'; '.join(errors)}", "warning")
|
||||
|
||||
if not media_ids:
|
||||
# No photo made it; announce the note as it is (v2.10.1)
|
||||
from starpunk.notes import _websub_changed
|
||||
_websub_changed(note)
|
||||
|
||||
# Crosspost after media is attached so images go out too (v2.1.0)
|
||||
if "syndication_form" in request.form:
|
||||
from starpunk.syndication import select_targets
|
||||
|
||||
@@ -11,6 +11,7 @@ from datetime import datetime, timedelta
|
||||
from flask import Blueprint, abort, render_template, Response, current_app, request, send_from_directory
|
||||
|
||||
from starpunk.notes import list_notes, get_note
|
||||
from starpunk.websub import embed_hub
|
||||
from starpunk.feed import generate_feed_streaming # Legacy RSS
|
||||
from starpunk.feeds import (
|
||||
generate_rss,
|
||||
@@ -55,8 +56,14 @@ def _get_cached_notes():
|
||||
cache_duration = timedelta(seconds=cache_seconds)
|
||||
now = datetime.utcnow()
|
||||
|
||||
# The cached list is stale once any public note changed, in any process
|
||||
# (v2.10.0): every change bumps content_version in the database
|
||||
from starpunk.database import get_db
|
||||
from starpunk.websub import get_content_version
|
||||
version = (str(current_app.config["DATABASE_PATH"]), get_content_version(get_db(current_app)))
|
||||
|
||||
# Check if note list cache is valid
|
||||
if _feed_cache["notes"] and _feed_cache["timestamp"]:
|
||||
if _feed_cache["notes"] and _feed_cache["timestamp"] and _feed_cache.get("version") == version:
|
||||
cache_age = now - _feed_cache["timestamp"]
|
||||
if cache_age < cache_duration:
|
||||
# Use cached note list
|
||||
@@ -82,6 +89,7 @@ def _get_cached_notes():
|
||||
|
||||
_feed_cache["notes"] = notes
|
||||
_feed_cache["timestamp"] = now
|
||||
_feed_cache["version"] = version
|
||||
|
||||
return notes
|
||||
|
||||
@@ -122,6 +130,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
|
||||
notes=notes,
|
||||
limit=max_items,
|
||||
)
|
||||
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
|
||||
|
||||
# Create response with proper Content-Type including charset
|
||||
mime_type = get_mime_type(format_name)
|
||||
@@ -132,7 +141,12 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
|
||||
|
||||
# Caching enabled - use FeedCache
|
||||
feed_cache = get_cache()
|
||||
notes_checksum = feed_cache.generate_notes_checksum(notes)
|
||||
# The content version covers changes the note list doesn't show, such as
|
||||
# photos attached or captions edited (v2.10.1)
|
||||
from starpunk.database import get_db
|
||||
from starpunk.websub import get_content_version
|
||||
notes_checksum = (feed_cache.generate_notes_checksum(notes)
|
||||
+ ":" + get_content_version(get_db(current_app)))
|
||||
|
||||
# Check If-None-Match header for conditional requests
|
||||
if_none_match = request.headers.get('If-None-Match')
|
||||
@@ -170,6 +184,7 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
|
||||
notes=notes,
|
||||
limit=max_items,
|
||||
)
|
||||
content = embed_hub(format_name, content) # WebSub hub link (v2.10.0)
|
||||
|
||||
# Store in cache and get ETag
|
||||
etag = feed_cache.set(format_name, content, notes_checksum)
|
||||
@@ -498,15 +513,15 @@ def feed():
|
||||
)
|
||||
|
||||
# Route to appropriate generator
|
||||
if format_name == 'rss':
|
||||
return feed_rss()
|
||||
elif format_name == 'atom':
|
||||
return feed_atom()
|
||||
if format_name == 'atom':
|
||||
response = feed_atom()
|
||||
elif format_name == 'json':
|
||||
return feed_json()
|
||||
response = feed_json()
|
||||
else:
|
||||
# Shouldn't reach here, but be defensive
|
||||
return feed_rss()
|
||||
response = feed_rss()
|
||||
# The format (and its rel=self) depends on Accept (v2.10.1)
|
||||
response.headers.add('Vary', 'Accept')
|
||||
return response
|
||||
|
||||
|
||||
@bp.route("/feed.rss")
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
"""
|
||||
WebSub hub endpoint (v2.10.0, ADR-071)
|
||||
|
||||
POST /websub with hub.mode subscribe/unsubscribe, hub.topic, hub.callback,
|
||||
optional hub.secret and hub.lease_seconds. Answers 202 and verifies later
|
||||
(spec 5.1); errors are 4xx with a plain-text reason.
|
||||
"""
|
||||
|
||||
from flask import Blueprint, current_app, request
|
||||
|
||||
from starpunk.database import get_db
|
||||
from starpunk.rate_limiter import RateLimiter
|
||||
from starpunk import websub
|
||||
|
||||
bp = Blueprint("websub", __name__)
|
||||
|
||||
# Subscription requests per client IP per minute
|
||||
# (per gunicorn process; needs TRUSTED_PROXY=true behind a reverse proxy,
|
||||
# or every client shares the proxy's address)
|
||||
_limiter = RateLimiter(limit=30, window_seconds=60)
|
||||
_requests = 0
|
||||
|
||||
|
||||
def _text(message: str, status: int):
|
||||
return message + "\n", status, {"Content-Type": "text/plain; charset=utf-8"}
|
||||
|
||||
|
||||
@bp.route("/websub", methods=["POST"])
|
||||
def hub():
|
||||
if not websub.enabled():
|
||||
return _text("WebSub is not enabled on this site", 404)
|
||||
|
||||
from starpunk.routes.webmention import get_client_ip
|
||||
global _requests
|
||||
_requests += 1
|
||||
if _requests % 500 == 0:
|
||||
_limiter.cleanup() # forget idle clients so memory doesn't grow
|
||||
if not _limiter.is_allowed(get_client_ip()):
|
||||
return _text("Too many requests", 429)
|
||||
|
||||
# Media types are case-insensitive (v2.10.1)
|
||||
if request.mimetype != "application/x-www-form-urlencoded":
|
||||
return _text("Send an application/x-www-form-urlencoded body", 400)
|
||||
|
||||
try:
|
||||
websub.accept_subscription_request(request.form, get_db(current_app))
|
||||
except websub.SubscriptionError as e:
|
||||
return _text(str(e), e.status)
|
||||
|
||||
return _text("Accepted; the subscription will be verified", 202)
|
||||
|
||||
|
||||
@bp.route("/websub", methods=["GET"])
|
||||
def hub_info():
|
||||
return _text(
|
||||
"WebSub hub for this site's own feeds. Subscribe with a POST "
|
||||
"(https://www.w3.org/TR/websub/).", 200)
|
||||
+73
-11
@@ -16,8 +16,10 @@ References:
|
||||
|
||||
import ipaddress
|
||||
import socket
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
from urllib.parse import urljoin, urlparse
|
||||
|
||||
import httpx
|
||||
@@ -39,6 +41,11 @@ class SSRFBlockedError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ResolutionError(ValueError):
|
||||
"""The host name could not be resolved (may be temporary; v2.10.1)"""
|
||||
pass
|
||||
|
||||
|
||||
class FetchTooLargeError(Exception):
|
||||
"""Raised when a response is larger than the limit"""
|
||||
pass
|
||||
@@ -131,7 +138,7 @@ def validate_url_for_fetch(url: str) -> str:
|
||||
try:
|
||||
return safe_resolve(parsed.hostname)
|
||||
except socket.gaierror as e:
|
||||
raise ValueError(f"Could not resolve {parsed.hostname}: {e}")
|
||||
raise ResolutionError(f"Could not resolve {parsed.hostname}: {e}")
|
||||
|
||||
|
||||
def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
|
||||
@@ -157,18 +164,36 @@ def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
|
||||
)
|
||||
|
||||
|
||||
def safe_post(url: str, data: dict, headers: dict, timeout: float = TIMEOUT) -> httpx.Response:
|
||||
class _Watchdog:
|
||||
"""
|
||||
POST form data to a checked, pinned address. Redirects are not followed.
|
||||
Close a client when its deadline passes (v2.10.1)
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError, ValueError: URL is not allowed
|
||||
httpx.HTTPError: Network errors
|
||||
httpx timeouts apply per read, so a server sending one byte at a time
|
||||
could hold a request for hours; closing the client ends it.
|
||||
"""
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
|
||||
response = client.send(_pinned_request(client, 'POST', url, headers, data=data))
|
||||
response.read()
|
||||
return response
|
||||
|
||||
def __init__(self, client: httpx.Client, seconds: float):
|
||||
self.fired = False
|
||||
|
||||
def fire():
|
||||
self.fired = True
|
||||
try:
|
||||
client.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
self._timer = threading.Timer(max(seconds, 0.01), fire)
|
||||
self._timer.daemon = True
|
||||
|
||||
def __enter__(self):
|
||||
self._timer.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
self._timer.cancel()
|
||||
if self.fired and exc_type is not None and not isinstance(exc, httpx.TimeoutException):
|
||||
raise httpx.TimeoutException("Exceeded time limit") from exc
|
||||
return False
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -188,6 +213,42 @@ class FetchResult:
|
||||
return self.content.decode('utf-8', errors='replace')
|
||||
|
||||
|
||||
def safe_post(url: str, data: Optional[dict] = None, headers: Optional[dict] = None,
|
||||
timeout: float = TIMEOUT, content: Optional[bytes] = None,
|
||||
max_bytes: int = 4096) -> FetchResult:
|
||||
"""
|
||||
POST form data (or a raw body) to a checked, pinned address.
|
||||
|
||||
Redirects are not followed. `timeout` bounds the whole exchange, so a
|
||||
server trickling its response can't hold the caller (v2.10.1). Only the
|
||||
first max_bytes of the response are read (callers need the status and
|
||||
maybe an error message).
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError, ValueError: URL is not allowed
|
||||
httpx.HTTPError: Network errors
|
||||
"""
|
||||
kwargs = {'content': content} if content is not None else {'data': data}
|
||||
deadline = time.monotonic() + timeout
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client, \
|
||||
_Watchdog(client, timeout):
|
||||
request = _pinned_request(client, 'POST', url, headers or {}, **kwargs)
|
||||
response = client.send(request, stream=True)
|
||||
try:
|
||||
body = b''
|
||||
for chunk in response.iter_bytes(chunk_size=1024):
|
||||
body += chunk
|
||||
if len(body) >= max_bytes:
|
||||
body = body[:max_bytes]
|
||||
break
|
||||
if time.monotonic() > deadline:
|
||||
raise httpx.TimeoutException("Response exceeded time limit")
|
||||
return FetchResult(url=url, status_code=response.status_code, headers=response.headers,
|
||||
content=body, encoding=response.charset_encoding or 'utf-8')
|
||||
finally:
|
||||
response.close()
|
||||
|
||||
|
||||
def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
||||
max_redirects: int = MAX_REDIRECTS, timeout: float = TIMEOUT) -> FetchResult:
|
||||
"""
|
||||
@@ -210,7 +271,8 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
||||
"""
|
||||
deadline = time.monotonic() + timeout
|
||||
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client:
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client, \
|
||||
_Watchdog(client, timeout):
|
||||
for _ in range(max_redirects + 1):
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
|
||||
@@ -0,0 +1,443 @@
|
||||
"""
|
||||
WebSub publishing and the built-in hub (v2.10.0, ADR-071)
|
||||
|
||||
Publisher: topic responses advertise the hub and their own URL; a change to a
|
||||
public note clears cached feeds and queues delivery of every affected topic.
|
||||
|
||||
Hub: subscriptions to this site's own topics only. Requests are accepted
|
||||
here (202) and verified, delivered and retried by the WebSub worker.
|
||||
|
||||
References:
|
||||
- W3C WebSub: https://www.w3.org/TR/websub/
|
||||
- ADR-071: WebSub publishing with a built-in hub
|
||||
"""
|
||||
|
||||
import json
|
||||
import secrets
|
||||
from typing import Iterable, Optional
|
||||
from urllib.parse import urlparse
|
||||
|
||||
from flask import current_app, request
|
||||
|
||||
|
||||
# Topic endpoints and their fixed paths (tag pages are added per tag)
|
||||
FEED_TOPICS = {
|
||||
"public.feed_atom": "/feed.atom",
|
||||
"public.feed_rss": "/feed.rss",
|
||||
"public.feed_xml_legacy": "/feed.rss", # legacy alias of the RSS topic
|
||||
"public.feed_json": "/feed.json",
|
||||
}
|
||||
PAGE_TOPICS = {"public.index": "/", "public.likes": "/likes"}
|
||||
NEGOTIATED = { # /feed: the topic of whichever format was served
|
||||
"application/atom+xml": "/feed.atom",
|
||||
"application/rss+xml": "/feed.rss",
|
||||
"application/feed+json": "/feed.json",
|
||||
"application/json": "/feed.json",
|
||||
}
|
||||
|
||||
LEASE_DEFAULT = 10 * 24 * 3600
|
||||
LEASE_MIN = 24 * 3600
|
||||
LEASE_MAX = 30 * 24 * 3600
|
||||
MAX_SECRET_BYTES = 200
|
||||
|
||||
|
||||
class SubscriptionError(Exception):
|
||||
"""A subscription request the hub refuses (4xx with a plain-text reason)"""
|
||||
|
||||
def __init__(self, message: str, status: int = 400):
|
||||
super().__init__(message)
|
||||
self.status = status
|
||||
|
||||
|
||||
# --- Configuration -----------------------------------------------------------
|
||||
|
||||
def enabled() -> bool:
|
||||
return bool(current_app.config.get("WEBSUB_ENABLED", True))
|
||||
|
||||
|
||||
def external_hub() -> Optional[str]:
|
||||
"""URL of an external hub, if one is configured instead of the built-in one"""
|
||||
return current_app.config.get("WEBSUB_HUB_URL") or None
|
||||
|
||||
|
||||
def hub_url() -> str:
|
||||
# Built from SITE_URL so the worker can use it outside a request
|
||||
return external_hub() or site_url() + "/websub"
|
||||
|
||||
|
||||
def site_url() -> str:
|
||||
return current_app.config["SITE_URL"].rstrip("/")
|
||||
|
||||
|
||||
# --- Topics ------------------------------------------------------------------
|
||||
|
||||
def topic_url(path: str) -> str:
|
||||
return site_url() + path
|
||||
|
||||
|
||||
def tag_topic(tag_name: str) -> str:
|
||||
from urllib.parse import quote
|
||||
return f"{site_url()}/tag/{quote(tag_name, safe='')}"
|
||||
|
||||
|
||||
def topic_for_request(response=None) -> Optional[str]:
|
||||
"""The topic URL the current request serves, or None if it isn't a topic"""
|
||||
endpoint = request.endpoint
|
||||
if endpoint in PAGE_TOPICS:
|
||||
return topic_url(PAGE_TOPICS[endpoint])
|
||||
if endpoint in FEED_TOPICS:
|
||||
return topic_url(FEED_TOPICS[endpoint])
|
||||
if endpoint == "public.feed" and response is not None:
|
||||
mime = (response.mimetype or "").lower()
|
||||
return topic_url(NEGOTIATED[mime]) if mime in NEGOTIATED else None
|
||||
if endpoint == "public.tag":
|
||||
from starpunk.tags import normalize_tag
|
||||
tag = (request.view_args or {}).get("tag", "")
|
||||
return tag_topic(normalize_tag(tag)[0]) if tag else None
|
||||
return None
|
||||
|
||||
|
||||
_UNRESERVED = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")
|
||||
|
||||
|
||||
def decode_unreserved(value: str) -> str:
|
||||
"""
|
||||
Decode percent-encoded unreserved characters (%66 -> f), leaving the rest
|
||||
|
||||
Spec 5.1.1: hubs MUST always decode non-reserved characters in URL
|
||||
parameters, so /%66eed.atom and /feed.atom are the same topic.
|
||||
"""
|
||||
out, i = [], 0
|
||||
while i < len(value):
|
||||
if value[i] == "%" and i + 3 <= len(value):
|
||||
try:
|
||||
char = chr(int(value[i + 1:i + 3], 16))
|
||||
except ValueError:
|
||||
char = None
|
||||
if char is not None and char in _UNRESERVED:
|
||||
out.append(char)
|
||||
i += 3
|
||||
continue
|
||||
out.append(value[i])
|
||||
i += 1
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def canonical_topic(url: str) -> Optional[str]:
|
||||
"""
|
||||
A subscription's hub.topic as one of this site's topics, or None
|
||||
|
||||
Unreserved percent-encodings are decoded and the scheme and host are
|
||||
compared case-insensitively. Tag topics must name an existing tag.
|
||||
"""
|
||||
if not url:
|
||||
return None
|
||||
site = urlparse(site_url())
|
||||
parsed = urlparse(decode_unreserved(url.strip()))
|
||||
if (parsed.scheme.lower(), parsed.netloc.lower()) != (site.scheme, site.netloc.lower()):
|
||||
return None
|
||||
path = parsed.path or "/"
|
||||
if parsed.query or parsed.fragment:
|
||||
return None
|
||||
if path in set(PAGE_TOPICS.values()) | set(FEED_TOPICS.values()):
|
||||
return topic_url(path)
|
||||
if path.startswith("/tag/"):
|
||||
from urllib.parse import unquote
|
||||
from starpunk.tags import get_tag_by_name, normalize_tag
|
||||
name = normalize_tag(unquote(path[len("/tag/"):]))[0]
|
||||
if name and get_tag_by_name(name):
|
||||
return tag_topic(name)
|
||||
return None
|
||||
|
||||
|
||||
def link_header_values(topic: str) -> list[str]:
|
||||
return [f'<{hub_url()}>; rel="hub"', f'<{topic}>; rel="self"']
|
||||
|
||||
|
||||
def add_discovery_headers(response):
|
||||
"""after_request: Link rel=hub and rel=self on topic responses (spec 4)"""
|
||||
if not enabled() or response.status_code not in (200, 304):
|
||||
return response
|
||||
topic = topic_for_request(response)
|
||||
if topic:
|
||||
for value in link_header_values(topic):
|
||||
response.headers.add("Link", value)
|
||||
return response
|
||||
|
||||
|
||||
def template_links() -> Optional[dict]:
|
||||
"""For base.html: {'hub', 'self'} on topic pages"""
|
||||
if not enabled() or not request:
|
||||
return None
|
||||
topic = topic_for_request()
|
||||
return {"hub": hub_url(), "self": topic} if topic else None
|
||||
|
||||
|
||||
def embed_hub(format_name: str, content: str) -> str:
|
||||
"""Add the hub link inside a generated feed, next to its self link"""
|
||||
if not enabled():
|
||||
return content
|
||||
hub = hub_url()
|
||||
if format_name == "atom":
|
||||
marker = '<link rel="self"'
|
||||
return content.replace(marker, f'<link rel="hub" href="{_xml(hub)}"/>\n {marker}', 1)
|
||||
if format_name == "rss":
|
||||
marker = "<atom:link "
|
||||
return content.replace(marker, f'<atom:link href="{_xml(hub)}" rel="hub"/>\n {marker}', 1)
|
||||
if format_name == "json":
|
||||
feed = json.loads(content)
|
||||
feed["hubs"] = [{"type": "WebSub", "url": hub}]
|
||||
return json.dumps(feed, ensure_ascii=False, indent=2)
|
||||
return content
|
||||
|
||||
|
||||
def _xml(value: str) -> str:
|
||||
return (value.replace("&", "&").replace("<", "<")
|
||||
.replace(">", ">").replace('"', """))
|
||||
|
||||
|
||||
# --- Changes -------------------------------------------------------------------
|
||||
|
||||
def get_content_version(db) -> str:
|
||||
row = db.execute("SELECT value FROM site_state WHERE key = 'content_version'").fetchone()
|
||||
return row[0] if row else "0"
|
||||
|
||||
|
||||
def bump_content_version(db) -> None:
|
||||
db.execute(
|
||||
"INSERT INTO site_state (key, value) VALUES ('content_version', '1') "
|
||||
"ON CONFLICT(key) DO UPDATE SET value = CAST(value AS INTEGER) + 1"
|
||||
)
|
||||
|
||||
|
||||
def note_changed(note, was_public: bool = False, old_tags: Iterable[str] = ()) -> None:
|
||||
"""
|
||||
A note was created, edited, deleted, restored or (un)published
|
||||
|
||||
Clears cached feeds in every process and queues delivery of the topics
|
||||
that show it. Changes that are never public (a draft edited) do nothing.
|
||||
|
||||
Args:
|
||||
note: The note (after the change; for a delete, before it)
|
||||
was_public: Whether it was public before the change
|
||||
old_tags: Its tag names before the change
|
||||
"""
|
||||
from starpunk.database import get_db
|
||||
|
||||
is_public = bool(note.published) and not getattr(note, "deleted_at", None)
|
||||
if not (is_public or was_public):
|
||||
return
|
||||
|
||||
db = get_db(current_app)
|
||||
bump_content_version(db)
|
||||
db.commit()
|
||||
|
||||
if not enabled():
|
||||
return
|
||||
|
||||
from starpunk.tags import get_note_tags
|
||||
try:
|
||||
tags = {t["name"] for t in get_note_tags(note.id)}
|
||||
except Exception:
|
||||
tags = set()
|
||||
tags |= set(old_tags)
|
||||
|
||||
# Likes appear on /likes, not on the home page or in the feeds
|
||||
if getattr(note, "like_of", None):
|
||||
topics = [topic_url("/likes")]
|
||||
else:
|
||||
topics = [topic_url(p) for p in ("/", "/feed.atom", "/feed.rss", "/feed.json")]
|
||||
topics += [tag_topic(name) for name in sorted(tags)]
|
||||
queue_topics(db, topics)
|
||||
|
||||
|
||||
def queue_topics(db, topics: Iterable[str]) -> int:
|
||||
"""Queue delivery of topics: to their subscribers, or a ping to an external hub"""
|
||||
topics = list(dict.fromkeys(topics))
|
||||
if not topics:
|
||||
return 0
|
||||
if external_hub():
|
||||
for topic in topics:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NULL AND topic = ?", (topic,))
|
||||
db.execute("INSERT INTO websub_deliveries (topic) VALUES (?)", (topic,))
|
||||
db.commit()
|
||||
return len(topics)
|
||||
|
||||
placeholders = ",".join("?" * len(topics))
|
||||
rows = db.execute(
|
||||
f"SELECT id, topic FROM websub_subscriptions WHERE topic IN ({placeholders}) "
|
||||
"AND verified_at IS NOT NULL AND expires_at > datetime('now')",
|
||||
topics,
|
||||
).fetchall()
|
||||
for sub_id, topic in rows:
|
||||
# The newest content replaces anything still waiting (ADR-071 6)
|
||||
db.execute("DELETE FROM websub_deliveries WHERE subscription_id = ?", (sub_id,))
|
||||
db.execute("INSERT INTO websub_deliveries (subscription_id, topic) VALUES (?, ?)", (sub_id, topic))
|
||||
db.commit()
|
||||
return len(rows)
|
||||
|
||||
|
||||
# --- Hub: subscription requests (spec 5.1) -------------------------------------
|
||||
|
||||
MAX_CALLBACK_LENGTH = 2048
|
||||
MAX_PENDING_INTENTS = 200
|
||||
ALLOWED_PORTS = {80, 443} # plus any port from 1024 up
|
||||
|
||||
|
||||
def _clean_callback(callback: str) -> str:
|
||||
"""
|
||||
Check a hub.callback and return it with an ASCII (IDNA) host
|
||||
|
||||
Raises:
|
||||
SubscriptionError: if it is not an acceptable callback URL
|
||||
"""
|
||||
import httpx
|
||||
|
||||
if not callback or len(callback) > MAX_CALLBACK_LENGTH:
|
||||
raise SubscriptionError(f"hub.callback must be a URL of at most {MAX_CALLBACK_LENGTH} characters")
|
||||
if any(ord(c) < 0x20 or ord(c) == 0x7f for c in callback) or " " in callback:
|
||||
raise SubscriptionError("hub.callback contains spaces or control characters")
|
||||
parsed = urlparse(callback)
|
||||
if parsed.scheme.lower() not in ("http", "https") or not parsed.hostname:
|
||||
raise SubscriptionError("hub.callback must be an absolute http(s) URL")
|
||||
try:
|
||||
port = parsed.port
|
||||
except ValueError:
|
||||
raise SubscriptionError("hub.callback has an invalid port")
|
||||
if port is not None and port not in ALLOWED_PORTS and port < 1024:
|
||||
raise SubscriptionError("hub.callback may not use that port")
|
||||
|
||||
try:
|
||||
host = parsed.hostname.encode("idna").decode("ascii")
|
||||
except UnicodeError:
|
||||
raise SubscriptionError("hub.callback has an invalid host name")
|
||||
netloc = host + (f":{port}" if port is not None else "")
|
||||
callback = parsed._replace(scheme=parsed.scheme.lower(), netloc=netloc).geturl()
|
||||
|
||||
if host.lower() == (urlparse(site_url()).hostname or "").lower():
|
||||
raise SubscriptionError("hub.callback may not be on this site")
|
||||
try:
|
||||
httpx.URL(callback)
|
||||
except Exception:
|
||||
raise SubscriptionError("hub.callback is not a valid URL")
|
||||
return callback
|
||||
|
||||
|
||||
def accept_subscription_request(form, db) -> None:
|
||||
"""
|
||||
Validate a subscription request and queue it for verification
|
||||
|
||||
Raises:
|
||||
SubscriptionError: with the status and plain-text reason to return
|
||||
"""
|
||||
from starpunk.webmention_ssrf import SSRFBlockedError, validate_url_for_fetch
|
||||
|
||||
mode = (form.get("hub.mode") or "").strip()
|
||||
if mode not in ("subscribe", "unsubscribe"):
|
||||
raise SubscriptionError("hub.mode must be subscribe or unsubscribe")
|
||||
if mode == "subscribe" and external_hub():
|
||||
# Unsubscribes are still taken, so old subscribers can leave
|
||||
raise SubscriptionError("This site uses another hub: " + external_hub(), 400)
|
||||
|
||||
callback = _clean_callback((form.get("hub.callback") or "").strip())
|
||||
try:
|
||||
callback_ip = validate_url_for_fetch(callback)
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
raise SubscriptionError(f"hub.callback is not reachable: {e}")
|
||||
|
||||
requested_topic = (form.get("hub.topic") or "").strip()
|
||||
topic = canonical_topic(requested_topic)
|
||||
if not topic:
|
||||
raise SubscriptionError("hub.topic is not a topic of this site; use the rel=self URL")
|
||||
|
||||
secret, lease = None, None
|
||||
if mode == "subscribe":
|
||||
# hub.secret and hub.lease_seconds MUST be ignored for unsubscriptions
|
||||
secret = form.get("hub.secret")
|
||||
if secret is not None:
|
||||
if secret == "":
|
||||
raise SubscriptionError("hub.secret may not be empty")
|
||||
if len(secret.encode("utf-8")) >= MAX_SECRET_BYTES:
|
||||
raise SubscriptionError("hub.secret must be less than 200 bytes")
|
||||
lease = LEASE_DEFAULT
|
||||
if form.get("hub.lease_seconds"):
|
||||
try:
|
||||
lease = int(form["hub.lease_seconds"])
|
||||
except ValueError:
|
||||
raise SubscriptionError("hub.lease_seconds must be a number")
|
||||
lease = max(LEASE_MIN, min(lease, LEASE_MAX))
|
||||
_check_limits(db, topic, callback, callback_ip)
|
||||
|
||||
# Each request gets its own id; the latest replaces any pending one
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO websub_intents
|
||||
(topic, callback, mode, secret, lease_seconds, request_id, requested_topic, callback_ip)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(topic, callback) DO UPDATE SET
|
||||
mode = excluded.mode, secret = excluded.secret,
|
||||
lease_seconds = excluded.lease_seconds, request_id = excluded.request_id,
|
||||
requested_topic = excluded.requested_topic, callback_ip = excluded.callback_ip,
|
||||
attempts = 0, next_attempt_at = CURRENT_TIMESTAMP, created_at = CURRENT_TIMESTAMP
|
||||
""",
|
||||
(topic, callback, mode, secret, lease, secrets.token_hex(16), requested_topic, callback_ip),
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _check_limits(db, topic: str, callback: str, callback_ip: str) -> None:
|
||||
"""
|
||||
Subscription limits, counting pending requests too (ADR-071 4)
|
||||
|
||||
Per callback address rather than host name, so many subdomains on one
|
||||
server share a limit. Renewing an existing subscription is always fine.
|
||||
"""
|
||||
if db.execute("SELECT 1 FROM websub_subscriptions WHERE topic = ? AND callback = ?",
|
||||
(topic, callback)).fetchone():
|
||||
return
|
||||
|
||||
pending = db.execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0]
|
||||
if pending >= MAX_PENDING_INTENTS:
|
||||
raise SubscriptionError("Too many subscription requests waiting; try again later", 429)
|
||||
|
||||
total = db.execute(
|
||||
"""
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT topic, callback FROM websub_subscriptions
|
||||
UNION
|
||||
SELECT topic, callback FROM websub_intents WHERE mode = 'subscribe'
|
||||
)
|
||||
"""
|
||||
).fetchone()[0]
|
||||
if total >= current_app.config.get("WEBSUB_MAX_SUBSCRIPTIONS", 1000):
|
||||
raise SubscriptionError("This hub has reached its subscription limit", 429)
|
||||
|
||||
per_address = db.execute(
|
||||
"""
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT topic, callback FROM websub_subscriptions WHERE callback_ip = ?
|
||||
UNION
|
||||
SELECT topic, callback FROM websub_intents WHERE mode = 'subscribe' AND callback_ip = ?
|
||||
)
|
||||
""",
|
||||
(callback_ip, callback_ip),
|
||||
).fetchone()[0]
|
||||
if per_address >= current_app.config.get("WEBSUB_MAX_PER_HOST", 25):
|
||||
raise SubscriptionError("Too many subscriptions for that callback's server", 429)
|
||||
|
||||
|
||||
def stats(db) -> dict:
|
||||
"""Active subscribers per topic and deliveries being retried (admin)"""
|
||||
by_topic = db.execute(
|
||||
"SELECT topic, COUNT(*) FROM websub_subscriptions WHERE expires_at > datetime('now') "
|
||||
"GROUP BY topic ORDER BY COUNT(*) DESC, topic"
|
||||
).fetchall()
|
||||
retrying = db.execute(
|
||||
"SELECT COUNT(*) FROM websub_deliveries WHERE attempts > 0"
|
||||
).fetchone()[0]
|
||||
return {"by_topic": [(t, n) for t, n in by_topic], "retrying": retrying,
|
||||
"external": external_hub()}
|
||||
|
||||
|
||||
def new_challenge() -> str:
|
||||
return secrets.token_urlsafe(32)
|
||||
@@ -0,0 +1,362 @@
|
||||
"""
|
||||
WebSub worker: verification of intent and content distribution (v2.10.0)
|
||||
|
||||
Runs like the Webmention worker: a daemon thread in every gunicorn process,
|
||||
but only the process holding the file lock works the queues.
|
||||
|
||||
- Intents (spec 5.3): GET the callback with a challenge; a 2xx echoing it
|
||||
verifies. Anything else (including redirects) leaves the subscription as
|
||||
it was. Network errors, DNS failures and 5xx are retried a few times.
|
||||
- Deliveries (spec 7): render the topic in-app and POST it with
|
||||
Link rel=hub/self and, if there is a secret, X-Hub-Signature (sha256).
|
||||
Failures are retried; the subscription stays until its lease ends.
|
||||
- External hub: a delivery without a subscription is a publish ping.
|
||||
|
||||
v2.10.1: every queue item is handled on its own, so one bad item can't stop
|
||||
the worker; all outbound requests have a hard time limit.
|
||||
|
||||
References:
|
||||
- https://www.w3.org/TR/websub/
|
||||
- ADR-071
|
||||
"""
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import hmac
|
||||
import logging
|
||||
import threading
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from urllib.parse import urlencode, urlparse, urlsplit, urlunsplit
|
||||
|
||||
import httpx
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
DELIVERY_BACKOFF = [60, 300, 1800, 7200, 21600] # then give up on this delivery
|
||||
INTENT_BACKOFF = [60, 300, 900]
|
||||
PRUNE_EVERY = timedelta(hours=1)
|
||||
BATCH = 10
|
||||
VERIFY_MAX_BYTES = 4096 # a challenge reply is short; larger means not verified
|
||||
|
||||
|
||||
def callback_with_params(callback: str, params: dict) -> str:
|
||||
"""Append parameters to the callback, keeping its query string (spec 5.3)"""
|
||||
parts = urlsplit(callback)
|
||||
query = parts.query + ("&" if parts.query else "") + urlencode(params)
|
||||
return urlunsplit((parts.scheme, parts.netloc, parts.path, query, parts.fragment))
|
||||
|
||||
|
||||
def signature(secret: str, body: bytes) -> str:
|
||||
return "sha256=" + hmac.new(secret.encode("utf-8"), body, hashlib.sha256).hexdigest()
|
||||
|
||||
|
||||
def _now() -> str:
|
||||
return datetime.utcnow().strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
|
||||
def _later(seconds: int) -> str:
|
||||
return (datetime.utcnow() + timedelta(seconds=seconds)).strftime("%Y-%m-%d %H:%M:%S")
|
||||
|
||||
|
||||
class WebSubWorker(threading.Thread):
|
||||
"""Daemon thread that verifies subscriptions and delivers content"""
|
||||
|
||||
def __init__(self, app, poll_interval: int = 5):
|
||||
super().__init__(daemon=True, name="websub-worker")
|
||||
self.app = app
|
||||
self.poll_interval = poll_interval
|
||||
self._stop_event = threading.Event()
|
||||
self._lock_file = None
|
||||
self._last_prune = None
|
||||
|
||||
# --- thread plumbing ----------------------------------------------------
|
||||
|
||||
def run(self):
|
||||
logger.info("WebSub worker started")
|
||||
while not self._stop_event.is_set():
|
||||
if not self._acquire_lock():
|
||||
self._stop_event.wait(self.poll_interval)
|
||||
continue
|
||||
try:
|
||||
with self.app.app_context():
|
||||
done = self.process_once()
|
||||
except Exception as e:
|
||||
logger.error(f"WebSub worker error: {e}", exc_info=True)
|
||||
done = 0
|
||||
if not done:
|
||||
self._stop_event.wait(self.poll_interval)
|
||||
self._release_lock()
|
||||
|
||||
def stop(self):
|
||||
self._stop_event.set()
|
||||
|
||||
def _acquire_lock(self) -> bool:
|
||||
if self._lock_file:
|
||||
return True
|
||||
path = Path(self.app.config["DATABASE_PATH"]).parent / "websub-worker.lock"
|
||||
lock_file = open(path, "w")
|
||||
try:
|
||||
fcntl.flock(lock_file, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except OSError:
|
||||
lock_file.close()
|
||||
return False
|
||||
self._lock_file = lock_file
|
||||
logger.info("WebSub worker holds the queue lock")
|
||||
return True
|
||||
|
||||
def _release_lock(self):
|
||||
if self._lock_file:
|
||||
self._lock_file.close()
|
||||
self._lock_file = None
|
||||
|
||||
# --- one pass (also called directly by tests) ---------------------------
|
||||
|
||||
def process_once(self) -> int:
|
||||
from starpunk.database import get_db
|
||||
|
||||
db = get_db(self.app)
|
||||
done = 0
|
||||
rendered = {} # topic -> (status, content type, body), once per pass
|
||||
|
||||
for row in db.execute(
|
||||
"SELECT * FROM websub_intents WHERE next_attempt_at <= ? ORDER BY next_attempt_at, id LIMIT ?",
|
||||
(_now(), BATCH),
|
||||
).fetchall():
|
||||
self._guarded(db, row, "verification", self.verify, self._intent_crashed)
|
||||
done += 1
|
||||
for row in db.execute(
|
||||
"SELECT * FROM websub_deliveries WHERE next_attempt_at <= ? ORDER BY next_attempt_at, id LIMIT ?",
|
||||
(_now(), BATCH),
|
||||
).fetchall():
|
||||
self._guarded(db, row, "delivery", lambda d, r: self.deliver(d, r, rendered), self._retry_or_drop)
|
||||
done += 1
|
||||
if self._last_prune is None or datetime.utcnow() - self._last_prune > PRUNE_EVERY:
|
||||
self.prune(db)
|
||||
return done
|
||||
|
||||
def _guarded(self, db, row, what, handler, on_crash) -> None:
|
||||
"""Handle one queue item; an unexpected error affects only that item"""
|
||||
try:
|
||||
handler(db, row)
|
||||
except Exception as e:
|
||||
logger.error(f"WebSub {what} {row['id']} failed unexpectedly: {e}", exc_info=True)
|
||||
try:
|
||||
db.rollback()
|
||||
on_crash(db, row, f"{type(e).__name__}: {e}"[:200])
|
||||
db.commit()
|
||||
except Exception as cleanup:
|
||||
logger.error(f"WebSub could not record the failure: {cleanup}")
|
||||
|
||||
def _intent_crashed(self, db, intent, error) -> None:
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
|
||||
(intent["id"], intent["request_id"]))
|
||||
|
||||
# --- verification of intent (spec 5.3) ----------------------------------
|
||||
|
||||
def verify(self, db, intent) -> None:
|
||||
from starpunk.websub import new_challenge
|
||||
from starpunk.webmention_ssrf import (
|
||||
FetchTooLargeError, ResolutionError, SSRFBlockedError, safe_get, user_agent,
|
||||
)
|
||||
|
||||
challenge = new_challenge()
|
||||
# The topic exactly as the subscriber sent it (spec 5.3)
|
||||
params = {"hub.mode": intent["mode"],
|
||||
"hub.topic": intent["requested_topic"] or intent["topic"],
|
||||
"hub.challenge": challenge}
|
||||
if intent["mode"] == "subscribe":
|
||||
params["hub.lease_seconds"] = intent["lease_seconds"]
|
||||
url = callback_with_params(intent["callback"], params)
|
||||
|
||||
retry = False
|
||||
try:
|
||||
result = safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
|
||||
max_redirects=0, max_bytes=VERIFY_MAX_BYTES)
|
||||
# Exact match; a trailing newline from a framework is tolerated
|
||||
verified = 200 <= result.status_code < 300 and result.text.rstrip("\r\n") == challenge
|
||||
retry = result.status_code >= 500
|
||||
error = None if verified else f"HTTP {result.status_code}" + (
|
||||
"" if result.status_code >= 300 else ": challenge not echoed")
|
||||
except ResolutionError as e:
|
||||
verified, retry, error = False, True, str(e)
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
verified, error = False, f"Blocked: {e}"
|
||||
except FetchTooLargeError:
|
||||
verified, error = False, "Reply too large to be a challenge"
|
||||
except httpx.TooManyRedirects:
|
||||
verified, error = False, "Redirected (not allowed)"
|
||||
except httpx.HTTPError as e:
|
||||
verified, retry, error = False, True, (str(e) or type(e).__name__)[:200]
|
||||
|
||||
# Only act if no newer request replaced this one meanwhile
|
||||
current = db.execute("SELECT request_id FROM websub_intents WHERE id = ?", (intent["id"],)).fetchone()
|
||||
if current is None or current[0] != intent["request_id"]:
|
||||
return
|
||||
|
||||
if verified:
|
||||
if intent["mode"] == "subscribe":
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO websub_subscriptions
|
||||
(topic, callback, secret, lease_seconds, expires_at, verified_at, callback_ip)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(topic, callback) DO UPDATE SET
|
||||
secret = excluded.secret, lease_seconds = excluded.lease_seconds,
|
||||
expires_at = excluded.expires_at, verified_at = excluded.verified_at,
|
||||
callback_ip = excluded.callback_ip, last_error = NULL
|
||||
""",
|
||||
(intent["topic"], intent["callback"], intent["secret"], intent["lease_seconds"],
|
||||
_later(intent["lease_seconds"]), _now(), intent["callback_ip"]),
|
||||
)
|
||||
else:
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE topic = ? AND callback = ?",
|
||||
(intent["topic"], intent["callback"]))
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
|
||||
(intent["id"], intent["request_id"]))
|
||||
logger.info(f"WebSub {intent['mode']} verified: {intent['callback']} -> {intent['topic']}")
|
||||
elif retry and intent["attempts"] < len(INTENT_BACKOFF):
|
||||
db.execute(
|
||||
"UPDATE websub_intents SET attempts = attempts + 1, next_attempt_at = ? "
|
||||
"WHERE id = ? AND request_id IS ?",
|
||||
(_later(INTENT_BACKOFF[intent["attempts"]]), intent["id"], intent["request_id"]),
|
||||
)
|
||||
else:
|
||||
# Verification failed: the subscription state stays as it was
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
|
||||
(intent["id"], intent["request_id"]))
|
||||
logger.info(f"WebSub {intent['mode']} not verified for {intent['callback']}: {error}")
|
||||
db.commit()
|
||||
|
||||
# --- content distribution (spec 7) ---------------------------------------
|
||||
|
||||
def render_topic(self, topic: str):
|
||||
"""(status, content_type, body) of a topic, rendered in-app"""
|
||||
path = urlparse(topic).path or "/"
|
||||
response = self.app.test_client().get(path, headers={"Accept": "*/*"})
|
||||
return response.status_code, response.headers.get("Content-Type", ""), response.get_data()
|
||||
|
||||
def deliver(self, db, delivery, rendered: Optional[dict] = None) -> None:
|
||||
from starpunk.websub import hub_url
|
||||
from starpunk.webmention_ssrf import ResolutionError, SSRFBlockedError, safe_post, user_agent
|
||||
|
||||
rendered = {} if rendered is None else rendered
|
||||
site = self.app.config["SITE_URL"]
|
||||
sub = None
|
||||
if delivery["subscription_id"] is None:
|
||||
# External hub: ping it (hub.url is the de facto parameter)
|
||||
target, kwargs = hub_url(), {
|
||||
"data": {"hub.mode": "publish", "hub.url": delivery["topic"], "hub.topic": delivery["topic"]},
|
||||
"headers": {"User-Agent": user_agent(site)},
|
||||
}
|
||||
else:
|
||||
sub = db.execute(
|
||||
"SELECT * FROM websub_subscriptions WHERE id = ? AND expires_at > ?",
|
||||
(delivery["subscription_id"], _now()),
|
||||
).fetchone()
|
||||
if sub is None: # unsubscribed or expired meanwhile
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
db.commit()
|
||||
return
|
||||
if sub["topic"] not in rendered:
|
||||
rendered[sub["topic"]] = self.render_topic(sub["topic"])
|
||||
status, content_type, body = rendered[sub["topic"]]
|
||||
if status in (404, 410):
|
||||
# The topic is gone: end the subscription and say so (spec 5.2)
|
||||
self._deny(sub, "The topic no longer exists")
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
db.commit()
|
||||
return
|
||||
if status != 200:
|
||||
self._retry_or_drop(db, delivery, f"Topic rendered HTTP {status}")
|
||||
db.commit()
|
||||
return
|
||||
headers = {
|
||||
"Content-Type": content_type,
|
||||
"Link": f'<{hub_url()}>; rel="hub", <{sub["topic"]}>; rel="self"',
|
||||
"User-Agent": user_agent(site),
|
||||
}
|
||||
if sub["secret"] is not None:
|
||||
headers["X-Hub-Signature"] = signature(sub["secret"], body)
|
||||
target, kwargs = sub["callback"], {"content": body, "headers": headers}
|
||||
|
||||
permanent = False
|
||||
try:
|
||||
result = safe_post(target, **kwargs)
|
||||
status, error = result.status_code, f"HTTP {result.status_code}"
|
||||
except ResolutionError as e:
|
||||
status, error = None, str(e) # DNS can recover: retry
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
status, error, permanent = None, f"Blocked: {e}", True
|
||||
except httpx.HTTPError as e:
|
||||
status, error = None, (str(e) or type(e).__name__)[:200]
|
||||
|
||||
succeeded = status is not None and 200 <= status < 300
|
||||
if succeeded:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
if sub is not None:
|
||||
db.execute("UPDATE websub_subscriptions SET last_success_at = ?, last_error = NULL WHERE id = ?",
|
||||
(_now(), sub["id"]))
|
||||
elif sub is not None and status == 410:
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
logger.info(f"WebSub subscriber gone (410), removed: {sub['callback']}")
|
||||
else:
|
||||
if permanent:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
logger.warning(f"WebSub delivery of {delivery['topic']} refused: {error}")
|
||||
else:
|
||||
self._retry_or_drop(db, delivery, error)
|
||||
if sub is not None:
|
||||
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
|
||||
db.commit()
|
||||
|
||||
def _retry_or_drop(self, db, delivery, error: str) -> None:
|
||||
"""
|
||||
Retry a failed delivery with backoff, then give up on it
|
||||
|
||||
The subscription itself stays active until its lease ends (spec 7);
|
||||
the next change is delivered again.
|
||||
"""
|
||||
if delivery["attempts"] < len(DELIVERY_BACKOFF):
|
||||
db.execute(
|
||||
"UPDATE websub_deliveries SET attempts = attempts + 1, next_attempt_at = ?, last_error = ? WHERE id = ?",
|
||||
(_later(DELIVERY_BACKOFF[delivery["attempts"]]), error, delivery["id"]),
|
||||
)
|
||||
else:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
logger.warning(f"WebSub delivery of {delivery['topic']} gave up: {error}")
|
||||
|
||||
def _deny(self, sub, reason: str) -> None:
|
||||
"""Tell a subscriber its subscription has ended (hub.mode=denied), best effort"""
|
||||
from starpunk.webmention_ssrf import safe_get, user_agent
|
||||
url = callback_with_params(sub["callback"], {"hub.mode": "denied", "hub.topic": sub["topic"],
|
||||
"hub.reason": reason})
|
||||
try:
|
||||
safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
|
||||
max_redirects=0, max_bytes=VERIFY_MAX_BYTES)
|
||||
except Exception as e:
|
||||
logger.info(f"WebSub denial to {sub['callback']} not delivered: {e}")
|
||||
|
||||
# --- housekeeping -------------------------------------------------------
|
||||
|
||||
def prune(self, db) -> int:
|
||||
"""Remove subscriptions whose lease has run out, and their deliveries"""
|
||||
cursor = db.execute("DELETE FROM websub_subscriptions WHERE expires_at <= ?", (_now(),))
|
||||
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND "
|
||||
"subscription_id NOT IN (SELECT id FROM websub_subscriptions)")
|
||||
db.commit()
|
||||
self._last_prune = datetime.utcnow()
|
||||
if cursor.rowcount:
|
||||
logger.info(f"WebSub: removed {cursor.rowcount} expired subscriptions")
|
||||
return cursor.rowcount
|
||||
|
||||
|
||||
def start_worker(app) -> Optional[WebSubWorker]:
|
||||
if not app.config.get("WEBSUB_ENABLED", True) or app.config.get("TESTING", False):
|
||||
return None
|
||||
worker = WebSubWorker(app, poll_interval=app.config.get("WEBSUB_POLL_INTERVAL", 5))
|
||||
worker.start()
|
||||
return worker
|
||||
@@ -20,6 +20,21 @@
|
||||
{% endif %}
|
||||
</div>
|
||||
|
||||
{# WebSub subscribers (v2.10.0) #}
|
||||
{% if websub_stats %}
|
||||
<p class="note-meta websub-stats">
|
||||
{% if websub_stats.external %}
|
||||
WebSub: using {{ websub_stats.external }}
|
||||
{% elif websub_stats.by_topic %}
|
||||
WebSub subscribers:
|
||||
{% for topic, count in websub_stats.by_topic %}{{ topic|replace(config.SITE_URL.rstrip('/'), '') or '/' }} {{ count }}{% if not loop.last %}, {% endif %}{% endfor %}
|
||||
{% if websub_stats.retrying %} · {{ websub_stats.retrying }} deliveries being retried{% endif %}
|
||||
{% else %}
|
||||
WebSub: no subscribers yet
|
||||
{% endif %}
|
||||
</p>
|
||||
{% endif %}
|
||||
|
||||
{% if notes %}
|
||||
<div class="table-scroll">
|
||||
<table class="note-table">
|
||||
|
||||
@@ -14,6 +14,13 @@
|
||||
<link rel="webmention" href="/webmention">
|
||||
{% endif %}
|
||||
|
||||
{# WebSub hub and topic on feed pages (v2.10.0) #}
|
||||
{% set websub = websub_links() %}
|
||||
{% if websub %}
|
||||
<link rel="hub" href="{{ websub.hub }}">
|
||||
<link rel="self" href="{{ websub.self }}">
|
||||
{% endif %}
|
||||
|
||||
{# Micropub and IndieAuth discovery (independent of Webmention, v2.8.0) #}
|
||||
<link rel="micropub" href="{{ absolute_url_for('micropub.micropub_endpoint') }}">
|
||||
<link rel="indieauth-metadata" href="{{ absolute_url_for('indieauth.metadata') }}">
|
||||
|
||||
+1
-1
@@ -230,7 +230,7 @@ class TestGenerateFeed:
|
||||
|
||||
# Check for Atom namespace and self-link
|
||||
assert "atom" in feed_xml
|
||||
assert "feed.xml" in feed_xml
|
||||
assert "feed.rss" in feed_xml # self link (v2.10.0: /feed.rss is the RSS topic)
|
||||
assert 'rel="self"' in feed_xml
|
||||
|
||||
def test_generate_feed_item_structure(self, app, sample_notes):
|
||||
|
||||
@@ -0,0 +1,318 @@
|
||||
"""
|
||||
WebSub publisher and built-in hub (v2.10.0, ADR-071)
|
||||
|
||||
Section numbers refer to the W3C WebSub Recommendation. Subscribers are
|
||||
simulated with an httpx MockTransport behind the SSRF-safe fetcher.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import sqlite3
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
import httpx
|
||||
import mf2py
|
||||
import pytest
|
||||
|
||||
from starpunk.websub_worker import WebSubWorker
|
||||
|
||||
SITE = "http://localhost:5000"
|
||||
HUB = f"{SITE}/websub"
|
||||
PUBLIC = [(2, 1, 6, "", ("93.184.216.34", 0))]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def web(monkeypatch):
|
||||
"""Subscriber callbacks: handler(request) -> response, by host"""
|
||||
hosts, seen = {}, []
|
||||
# Every test client shares one IP; don't let earlier tests rate-limit this one
|
||||
from starpunk.rate_limiter import RateLimiter
|
||||
monkeypatch.setattr("starpunk.routes.websub._limiter", RateLimiter(limit=1000, window_seconds=60))
|
||||
|
||||
def handler(request):
|
||||
host = request.headers["host"]
|
||||
seen.append(request)
|
||||
return hosts[host](request)
|
||||
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", lambda *a, **k: PUBLIC)
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.TRANSPORT", httpx.MockTransport(handler))
|
||||
return hosts, seen
|
||||
|
||||
|
||||
def echo(request):
|
||||
"""A subscriber that confirms every verification and accepts content"""
|
||||
if request.method == "GET":
|
||||
return httpx.Response(200, text=parse_qs(urlparse(str(request.url)).query)["hub.challenge"][0])
|
||||
return httpx.Response(202)
|
||||
|
||||
|
||||
def db(app):
|
||||
conn = sqlite3.connect(app.config["DATABASE_PATH"])
|
||||
conn.row_factory = sqlite3.Row
|
||||
return conn
|
||||
|
||||
|
||||
def run_worker(app, times=1):
|
||||
worker = WebSubWorker(app)
|
||||
with app.app_context():
|
||||
for _ in range(times):
|
||||
worker.process_once()
|
||||
return worker
|
||||
|
||||
|
||||
def subscribe(client, topic, callback="https://sub.example/cb?id=1", **extra):
|
||||
form = {"hub.mode": "subscribe", "hub.topic": topic, "hub.callback": callback, **extra}
|
||||
return client.post("/websub", data=form)
|
||||
|
||||
|
||||
def make_note(app, content="Hello", **kw):
|
||||
from starpunk.notes import create_note
|
||||
with app.test_request_context():
|
||||
return create_note(content, published=kw.pop("published", True), **kw)
|
||||
|
||||
|
||||
def links(response):
|
||||
return response.headers.getlist("Link")
|
||||
|
||||
|
||||
# 4 Discovery -------------------------------------------------------------------
|
||||
|
||||
@pytest.mark.parametrize("path,self_url", [
|
||||
("/", f"{SITE}/"),
|
||||
("/feed.atom", f"{SITE}/feed.atom"),
|
||||
("/feed.rss", f"{SITE}/feed.rss"),
|
||||
("/feed.xml", f"{SITE}/feed.rss"),
|
||||
("/feed.json", f"{SITE}/feed.json"),
|
||||
("/likes", f"{SITE}/likes"),
|
||||
])
|
||||
def test_topics_advertise_hub_and_self(client, path, self_url):
|
||||
response = client.get(path)
|
||||
values = links(response)
|
||||
assert f'<{HUB}>; rel="hub"' in values
|
||||
assert [v for v in values if 'rel="self"' in v] == [f'<{self_url}>; rel="self"']
|
||||
|
||||
|
||||
def test_negotiated_feed_self_is_the_format_served(client):
|
||||
response = client.get("/feed", headers={"Accept": "application/atom+xml"})
|
||||
assert f'<{SITE}/feed.atom>; rel="self"' in links(response)
|
||||
|
||||
|
||||
def test_tag_page_is_a_topic(app, client):
|
||||
make_note(app, tags=["Python"])
|
||||
assert f'<{SITE}/tag/python>; rel="self"' in links(client.get("/tag/Python"))
|
||||
|
||||
|
||||
def test_note_page_is_not_a_topic(app, client):
|
||||
note = make_note(app)
|
||||
assert not [v for v in links(client.get(f"/note/{note.slug}")) if 'rel="hub"' in v]
|
||||
|
||||
|
||||
def test_embedded_links(client):
|
||||
rels = mf2py.parse(doc=client.get("/").data.decode(), url=SITE + "/")["rels"]
|
||||
assert rels["hub"] == [HUB] and rels["self"] == [f"{SITE}/"]
|
||||
assert f'<link rel="hub" href="{HUB}"/>' in client.get("/feed.atom").data.decode()
|
||||
assert f'<atom:link href="{HUB}" rel="hub"/>' in client.get("/feed.rss").data.decode()
|
||||
assert json.loads(client.get("/feed.json").data)["hubs"] == [{"type": "WebSub", "url": HUB}]
|
||||
|
||||
|
||||
def test_external_hub(app, client, web):
|
||||
app.config["WEBSUB_HUB_URL"] = "https://hub.example/"
|
||||
hosts, seen = web
|
||||
hosts["hub.example"] = lambda r: httpx.Response(204)
|
||||
assert '<https://hub.example/>; rel="hub"' in links(client.get("/"))
|
||||
assert subscribe(client, f"{SITE}/").status_code == 400 # our hub is off
|
||||
|
||||
make_note(app)
|
||||
run_worker(app)
|
||||
pings = {parse_qs(r.content.decode())["hub.url"][0] for r in seen}
|
||||
assert pings == {f"{SITE}/", f"{SITE}/feed.atom", f"{SITE}/feed.rss", f"{SITE}/feed.json"}
|
||||
|
||||
|
||||
# 5.1 Subscription requests ----------------------------------------------------------
|
||||
|
||||
def test_subscription_request_is_accepted(app, client, web):
|
||||
response = subscribe(client, f"{SITE}/feed.atom", **{"hub.secret": "s3cret", "extra": "ignored"})
|
||||
assert response.status_code == 202
|
||||
row = db(app).execute("SELECT * FROM websub_intents").fetchone()
|
||||
assert (row["mode"], row["topic"], row["secret"]) == ("subscribe", f"{SITE}/feed.atom", "s3cret")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("form,reason", [
|
||||
({"hub.mode": "watch"}, "hub.mode"),
|
||||
({"hub.topic": "https://elsewhere.example/feed"}, "hub.topic"),
|
||||
({"hub.topic": f"{SITE}/note/x"}, "hub.topic"),
|
||||
({"hub.topic": f"{SITE}/tag/missing"}, "hub.topic"),
|
||||
({"hub.callback": "ftp://sub.example/cb"}, "hub.callback"),
|
||||
({"hub.secret": "x" * 200}, "hub.secret"),
|
||||
])
|
||||
def test_bad_requests(client, web, form, reason):
|
||||
base = {"hub.mode": "subscribe", "hub.topic": f"{SITE}/", "hub.callback": "https://sub.example/cb"}
|
||||
response = client.post("/websub", data={**base, **form})
|
||||
assert response.status_code == 400 and reason in response.data.decode()
|
||||
|
||||
|
||||
def test_private_callback_refused(client, monkeypatch):
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo",
|
||||
lambda *a, **k: [(2, 1, 6, "", ("10.0.0.2", 0))])
|
||||
assert subscribe(client, f"{SITE}/").status_code == 400
|
||||
|
||||
|
||||
def test_per_host_limit(app, client, web):
|
||||
app.config["WEBSUB_MAX_PER_HOST"] = 1
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = echo
|
||||
subscribe(client, f"{SITE}/")
|
||||
run_worker(app)
|
||||
assert subscribe(client, f"{SITE}/feed.atom").status_code == 429
|
||||
|
||||
|
||||
# 5.3 Verification of intent --------------------------------------------------------
|
||||
|
||||
def test_verification_activates_subscription(app, client, web):
|
||||
hosts, seen = web
|
||||
hosts["sub.example"] = echo
|
||||
subscribe(client, f"{SITE}/", **{"hub.lease_seconds": "86400"})
|
||||
run_worker(app)
|
||||
|
||||
query = parse_qs(urlparse(str(seen[0].url)).query)
|
||||
assert query["id"] == ["1"] # the callback's own query string is kept
|
||||
assert query["hub.mode"] == ["subscribe"] and query["hub.topic"] == [f"{SITE}/"]
|
||||
assert query["hub.lease_seconds"] == ["86400"]
|
||||
sub = db(app).execute("SELECT * FROM websub_subscriptions").fetchone()
|
||||
assert sub["topic"] == f"{SITE}/" and sub["verified_at"]
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0] == 0
|
||||
|
||||
|
||||
@pytest.mark.parametrize("response", [
|
||||
httpx.Response(200, text="wrong"),
|
||||
httpx.Response(404),
|
||||
httpx.Response(302, headers={"Location": "https://sub.example/other"}),
|
||||
])
|
||||
def test_failed_verification_changes_nothing(app, client, web, response):
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = lambda r: response
|
||||
subscribe(client, f"{SITE}/")
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_unsubscribe(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = echo
|
||||
subscribe(client, f"{SITE}/")
|
||||
run_worker(app)
|
||||
client.post("/websub", data={"hub.mode": "unsubscribe", "hub.topic": f"{SITE}/",
|
||||
"hub.callback": "https://sub.example/cb?id=1"})
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_resubscribe_changes_secret_only_once_verified(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = echo
|
||||
subscribe(client, f"{SITE}/", **{"hub.secret": "one"})
|
||||
run_worker(app)
|
||||
hosts["sub.example"] = lambda r: httpx.Response(404)
|
||||
subscribe(client, f"{SITE}/", **{"hub.secret": "two"})
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT secret FROM websub_subscriptions").fetchone()[0] == "one"
|
||||
|
||||
|
||||
# 6, 7 Content distribution ---------------------------------------------------------
|
||||
|
||||
def _subscribed(app, client, web, topic=f"{SITE}/feed.atom", secret=None):
|
||||
hosts, seen = web
|
||||
hosts["sub.example"] = echo
|
||||
extra = {"hub.secret": secret} if secret else {}
|
||||
subscribe(client, topic, **extra)
|
||||
run_worker(app)
|
||||
seen.clear()
|
||||
return hosts, seen
|
||||
|
||||
|
||||
def test_new_note_is_delivered_signed(app, client, web):
|
||||
hosts, seen = _subscribed(app, client, web, secret="s3cret")
|
||||
make_note(app, "Fresh post")
|
||||
run_worker(app)
|
||||
|
||||
(post,) = [r for r in seen if r.method == "POST"]
|
||||
assert post.headers["content-type"].startswith("application/atom+xml")
|
||||
assert b"Fresh post" in post.content
|
||||
assert f'<{HUB}>; rel="hub"' in post.headers["link"] and f'<{SITE}/feed.atom>; rel="self"' in post.headers["link"]
|
||||
expected = "sha256=" + hmac.new(b"s3cret", post.content, hashlib.sha256).hexdigest()
|
||||
assert post.headers["x-hub-signature"] == expected
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_no_signature_without_secret(app, client, web):
|
||||
hosts, seen = _subscribed(app, client, web)
|
||||
make_note(app)
|
||||
run_worker(app)
|
||||
assert "x-hub-signature" not in [r for r in seen if r.method == "POST"][0].headers
|
||||
|
||||
|
||||
def test_drafts_are_not_delivered(app, client, web):
|
||||
_subscribed(app, client, web)
|
||||
make_note(app, published=False)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_tag_topics_include_removed_tags(app, client, web):
|
||||
from starpunk.notes import update_note
|
||||
note = make_note(app, tags=["old"])
|
||||
_subscribed(app, client, web, topic=f"{SITE}/tag/old")
|
||||
with app.test_request_context():
|
||||
update_note(id=note.id, content="Hello", tags=["new"])
|
||||
assert db(app).execute("SELECT topic FROM websub_deliveries").fetchone()[0] == f"{SITE}/tag/old"
|
||||
|
||||
|
||||
def test_delete_is_delivered(app, client, web):
|
||||
from starpunk.notes import delete_note
|
||||
note = make_note(app, "Going away")
|
||||
hosts, seen = _subscribed(app, client, web)
|
||||
with app.test_request_context():
|
||||
delete_note(id=note.id, soft=False)
|
||||
run_worker(app)
|
||||
(post,) = [r for r in seen if r.method == "POST"]
|
||||
assert b"Going away" not in post.content
|
||||
|
||||
|
||||
def test_410_removes_subscription(app, client, web):
|
||||
hosts, _ = _subscribed(app, client, web)
|
||||
hosts["sub.example"] = lambda r: httpx.Response(410)
|
||||
make_note(app)
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_failures_are_retried(app, client, web):
|
||||
hosts, _ = _subscribed(app, client, web)
|
||||
hosts["sub.example"] = lambda r: httpx.Response(500)
|
||||
make_note(app)
|
||||
run_worker(app)
|
||||
row = db(app).execute("SELECT attempts, next_attempt_at, last_error FROM websub_deliveries").fetchone()
|
||||
assert row["attempts"] == 1 and row["last_error"] == "HTTP 500"
|
||||
|
||||
|
||||
def test_expired_subscriptions_get_nothing(app, client, web):
|
||||
_subscribed(app, client, web)
|
||||
conn = db(app)
|
||||
conn.execute("UPDATE websub_subscriptions SET expires_at = datetime('now', '-1 minute')")
|
||||
conn.commit()
|
||||
make_note(app)
|
||||
assert conn.execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
run_worker(app) # prunes
|
||||
assert conn.execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0] == 0
|
||||
|
||||
|
||||
# Feeds are fresh right after a change (ADR-071 3) ------------------------------------
|
||||
|
||||
def test_feed_shows_new_note_immediately(app, client):
|
||||
from starpunk.routes import public
|
||||
make_note(app, "First")
|
||||
assert b"First" in client.get("/feed.atom").data
|
||||
make_note(app, "Second") # cached note list is stale now
|
||||
assert b"Second" in client.get("/feed.atom").data
|
||||
assert public._feed_cache["notes"] is not None
|
||||
@@ -0,0 +1,316 @@
|
||||
"""
|
||||
WebSub hardening (v2.10.1): one test per finding of the v2.10.0 review
|
||||
"""
|
||||
|
||||
import io
|
||||
import socket
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
from unittest.mock import patch
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from tests.test_websub import ( # noqa: F401
|
||||
HUB, PUBLIC, SITE, db, echo, make_note, run_worker, subscribe, web,
|
||||
)
|
||||
from starpunk.websub_worker import WebSubWorker
|
||||
|
||||
|
||||
def jpeg(color="red"):
|
||||
buf = io.BytesIO()
|
||||
Image.new("RGB", (40, 30), color).save(buf, format="JPEG")
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def intents(app):
|
||||
return db(app).execute("SELECT * FROM websub_intents ORDER BY id").fetchall()
|
||||
|
||||
|
||||
def subs(app):
|
||||
return db(app).execute("SELECT * FROM websub_subscriptions ORDER BY id").fetchall()
|
||||
|
||||
|
||||
def subscribed(app, client, web, topic=f"{SITE}/feed.atom", callback="https://sub.example/cb"):
|
||||
hosts, seen = web
|
||||
hosts.setdefault(urlparse(callback).hostname, echo)
|
||||
subscribe(client, topic, callback=callback)
|
||||
run_worker(app)
|
||||
seen.clear()
|
||||
return hosts, seen
|
||||
|
||||
|
||||
# H1, H2: one bad item can't stop the worker -----------------------------------
|
||||
|
||||
def test_oversized_challenge_reply_does_not_wedge_worker(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["big.example"] = lambda r: httpx.Response(200, content=b"x" * 100_000)
|
||||
hosts["good.example"] = echo
|
||||
subscribe(client, f"{SITE}/", callback="https://big.example/cb")
|
||||
subscribe(client, f"{SITE}/", callback="https://good.example/cb")
|
||||
run_worker(app)
|
||||
assert [s["callback"] for s in subs(app)] == ["https://good.example/cb"]
|
||||
assert intents(app) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("callback", [
|
||||
"https://example.com/\x00", "https://example.com/a b", "https://example.com:25/cb",
|
||||
f"{SITE}/cb", "https://" + "a" * 2100 + ".example/",
|
||||
])
|
||||
def test_bad_callbacks_refused(client, web, callback):
|
||||
assert subscribe(client, f"{SITE}/", callback=callback).status_code == 400
|
||||
|
||||
|
||||
def test_unexpected_error_affects_only_that_item(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["good.example"] = echo
|
||||
|
||||
def crash(request):
|
||||
raise RuntimeError("something nobody expected")
|
||||
|
||||
hosts["bad.example"] = crash
|
||||
subscribe(client, f"{SITE}/", callback="https://bad.example/cb") # first in the queue
|
||||
subscribe(client, f"{SITE}/", callback="https://good.example/cb")
|
||||
run_worker(app)
|
||||
assert [s["callback"] for s in subs(app)] == ["https://good.example/cb"]
|
||||
assert intents(app) == []
|
||||
|
||||
|
||||
def test_idn_callback_is_punycoded(app, client, web):
|
||||
assert subscribe(client, f"{SITE}/", callback="https://bücher.example/cb").status_code == 202
|
||||
assert intents(app)[0]["callback"] == "https://xn--bcher-kva.example/cb"
|
||||
|
||||
|
||||
# H3, H4: photos are in what's delivered and in the feeds -----------------------
|
||||
|
||||
def test_new_photo_note_is_delivered_with_its_photo(app, client, web):
|
||||
from starpunk.media import attach_media_to_note, save_media
|
||||
hosts, seen = subscribed(app, client, web)
|
||||
with app.test_request_context():
|
||||
note = make_note(app, "With a photo", has_media=True)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
media = save_media(jpeg(), "cat.jpg")
|
||||
attach_media_to_note(note.id, [media["id"]], ["A cat"])
|
||||
run_worker(app)
|
||||
(post,) = [r for r in seen if r.method == "POST"]
|
||||
assert media["path"].encode() in post.content
|
||||
|
||||
|
||||
def test_photo_change_refreshes_feed_and_is_delivered(app, client, web):
|
||||
from starpunk.media import attach_media_to_note, save_media
|
||||
with app.test_request_context():
|
||||
note = make_note(app, "Photo later")
|
||||
first = save_media(jpeg("red"), "a.jpg")
|
||||
attach_media_to_note(note.id, [first["id"]], [""])
|
||||
assert first["path"].encode() in client.get("/feed.atom").data # cached now
|
||||
hosts, seen = subscribed(app, client, web)
|
||||
with app.test_request_context():
|
||||
second = save_media(jpeg("blue"), "b.jpg")
|
||||
attach_media_to_note(note.id, [second["id"]], [""])
|
||||
feed = client.get("/feed.atom").data
|
||||
assert second["path"].encode() in feed and first["path"].encode() not in feed
|
||||
run_worker(app)
|
||||
assert second["path"].encode() in [r for r in seen if r.method == "POST"][0].content
|
||||
|
||||
|
||||
# H5: a slow subscriber can't hold the worker ------------------------------------
|
||||
|
||||
def test_slow_callback_is_cut_off():
|
||||
from starpunk import webmention_ssrf as w
|
||||
|
||||
server = socket.socket()
|
||||
server.bind(("127.0.0.1", 0))
|
||||
server.listen(1)
|
||||
port = server.getsockname()[1]
|
||||
|
||||
def trickle():
|
||||
conn, _ = server.accept()
|
||||
conn.recv(65536)
|
||||
try:
|
||||
for b in b"HTTP/1.1 200 OK\r\nX-Slow: " + b"y" * 100:
|
||||
conn.send(bytes([b]))
|
||||
time.sleep(0.3)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
threading.Thread(target=trickle, daemon=True).start()
|
||||
start = time.monotonic()
|
||||
with patch.object(w, "safe_resolve", lambda host: "127.0.0.1"), pytest.raises(httpx.TimeoutException):
|
||||
w.safe_post(f"http://slow.example:{port}/", data={"a": "b"}, timeout=1)
|
||||
assert time.monotonic() - start < 3
|
||||
|
||||
|
||||
# Spec 7: subscriptions stay until their lease ends --------------------------------
|
||||
|
||||
def test_failing_subscriber_is_kept_until_lease_end(app, client, web):
|
||||
hosts, _ = subscribed(app, client, web)
|
||||
hosts["sub.example"] = lambda r: httpx.Response(500)
|
||||
conn = db(app)
|
||||
conn.execute("UPDATE websub_subscriptions SET verified_at = datetime('now', '-10 days')")
|
||||
conn.commit()
|
||||
make_note(app)
|
||||
conn.execute("UPDATE websub_deliveries SET attempts = 5") # last retry
|
||||
conn.commit()
|
||||
run_worker(app)
|
||||
assert conn.execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
assert len(subs(app)) == 1
|
||||
|
||||
|
||||
def test_dns_failure_is_retried(app, client, web, monkeypatch):
|
||||
subscribed(app, client, web)
|
||||
make_note(app)
|
||||
|
||||
def fail(*a, **k):
|
||||
raise socket.gaierror("temporary failure")
|
||||
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", fail)
|
||||
run_worker(app)
|
||||
row = db(app).execute("SELECT attempts, last_error FROM websub_deliveries").fetchone()
|
||||
assert row["attempts"] == 1 and "resolve" in row["last_error"]
|
||||
|
||||
|
||||
def test_render_error_is_retried(app, client, web):
|
||||
subscribed(app, client, web)
|
||||
make_note(app)
|
||||
with patch.object(WebSubWorker, "render_topic", lambda self, t: (500, "text/plain", b"")):
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT attempts FROM websub_deliveries").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_gone_topic_ends_subscription_with_denial(app, client, web):
|
||||
hosts, seen = subscribed(app, client, web)
|
||||
make_note(app)
|
||||
with patch.object(WebSubWorker, "render_topic", lambda self, t: (404, "text/html", b"")):
|
||||
run_worker(app)
|
||||
assert subs(app) == []
|
||||
denial = [r for r in seen if r.method == "GET"][0]
|
||||
assert parse_qs(urlparse(str(denial.url)).query)["hub.mode"] == ["denied"]
|
||||
|
||||
|
||||
# M1: limits count pending requests and the callback's address ----------------------
|
||||
|
||||
def test_limits_count_pending_requests(app, client, web):
|
||||
app.config["WEBSUB_MAX_PER_HOST"] = 3
|
||||
codes = [subscribe(client, f"{SITE}/", callback=f"https://s{i}.example/cb").status_code for i in range(5)]
|
||||
assert codes == [202, 202, 202, 429, 429] # all resolve to one address
|
||||
|
||||
|
||||
def test_renewal_is_allowed_at_the_limit(app, client, web):
|
||||
app.config["WEBSUB_MAX_PER_HOST"] = 1
|
||||
subscribed(app, client, web)
|
||||
assert subscribe(client, f"{SITE}/feed.atom", callback="https://sub.example/cb").status_code == 202
|
||||
|
||||
|
||||
# Spec 5.1 details -----------------------------------------------------------------
|
||||
|
||||
def test_unsubscribe_ignores_lease_and_secret(app, client, web):
|
||||
form = {"hub.mode": "unsubscribe", "hub.topic": f"{SITE}/", "hub.callback": "https://sub.example/cb",
|
||||
"hub.lease_seconds": "abc", "hub.secret": ""}
|
||||
assert client.post("/websub", data=form).status_code == 202
|
||||
|
||||
|
||||
def test_unreserved_characters_are_decoded(app, client, web):
|
||||
assert subscribe(client, f"{SITE}/%66eed%2Eatom").status_code == 202
|
||||
assert intents(app)[0]["topic"] == f"{SITE}/feed.atom"
|
||||
|
||||
|
||||
def test_verification_echoes_the_requested_topic(app, client, web):
|
||||
hosts, seen = web
|
||||
hosts["sub.example"] = echo
|
||||
make_note(app, tags=["Python"])
|
||||
subscribe(client, f"{SITE}/tag/Python")
|
||||
run_worker(app)
|
||||
assert parse_qs(urlparse(str(seen[0].url)).query)["hub.topic"] == [f"{SITE}/tag/Python"]
|
||||
assert subs(app)[0]["topic"] == f"{SITE}/tag/python"
|
||||
|
||||
|
||||
def test_content_type_is_case_insensitive(client, web):
|
||||
response = client.post("/websub", data="hub.mode=subscribe&hub.topic=" + SITE + "/&hub.callback=https://sub.example/cb",
|
||||
content_type="Application/X-WWW-Form-Urlencoded")
|
||||
assert response.status_code == 202
|
||||
|
||||
|
||||
def test_empty_secret_is_refused(client, web):
|
||||
assert subscribe(client, f"{SITE}/", **{"hub.secret": ""}).status_code == 400
|
||||
|
||||
|
||||
def test_challenge_must_match(app, client, web):
|
||||
hosts, _ = web
|
||||
|
||||
def padded(request):
|
||||
challenge = parse_qs(urlparse(str(request.url)).query)["hub.challenge"][0]
|
||||
return httpx.Response(200, text=f" {challenge} ")
|
||||
|
||||
hosts["sub.example"] = padded
|
||||
subscribe(client, f"{SITE}/")
|
||||
run_worker(app)
|
||||
assert subs(app) == []
|
||||
|
||||
|
||||
# L1: a newer request isn't lost while an older one is verified ------------------------
|
||||
|
||||
def test_newer_request_survives_older_verification(app, client, web):
|
||||
hosts, _ = web
|
||||
|
||||
def slow_echo(request):
|
||||
# A newer request (new secret) arrives while this one is being verified
|
||||
subscribe(client, f"{SITE}/", **{"hub.secret": "new"})
|
||||
return echo(request)
|
||||
|
||||
hosts["sub.example"] = slow_echo
|
||||
subscribe(client, f"{SITE}/", **{"hub.secret": "old"})
|
||||
run_worker(app)
|
||||
pending = intents(app)
|
||||
assert len(pending) == 1 and pending[0]["secret"] == "new"
|
||||
|
||||
|
||||
# L2: fewer renders and pushes ------------------------------------------------------
|
||||
|
||||
def test_topic_rendered_once_per_pass(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["a.example"] = hosts["b.example"] = echo
|
||||
for cb in ("https://a.example/cb", "https://b.example/cb"):
|
||||
subscribe(client, f"{SITE}/feed.atom", callback=cb)
|
||||
run_worker(app)
|
||||
make_note(app)
|
||||
calls = []
|
||||
original = WebSubWorker.render_topic
|
||||
|
||||
def counting(self, topic):
|
||||
calls.append(topic)
|
||||
return original(self, topic)
|
||||
|
||||
with patch.object(WebSubWorker, "render_topic", counting):
|
||||
run_worker(app)
|
||||
assert calls == [f"{SITE}/feed.atom"]
|
||||
|
||||
|
||||
def test_like_only_updates_likes_page(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = echo
|
||||
for topic in (f"{SITE}/", f"{SITE}/likes"):
|
||||
subscribe(client, topic)
|
||||
run_worker(app)
|
||||
make_note(app, "", like_of="https://elsewhere.example/post")
|
||||
topics = [r[0] for r in db(app).execute("SELECT topic FROM websub_deliveries")]
|
||||
assert topics == [f"{SITE}/likes"]
|
||||
|
||||
|
||||
# L5: feeds ----------------------------------------------------------------------
|
||||
|
||||
def test_etag_is_stable_across_processes(app, client):
|
||||
from starpunk.feeds import get_cache
|
||||
make_note(app)
|
||||
first = client.get("/feed.rss").headers["ETag"]
|
||||
get_cache().invalidate() # as if another process rendered it
|
||||
time.sleep(1.1) # a different build time
|
||||
assert client.get("/feed.rss").headers["ETag"] == first
|
||||
|
||||
|
||||
def test_negotiated_feed_varies_on_accept(client):
|
||||
assert "Accept" in client.get("/feed", headers={"Accept": "application/atom+xml"}).headers.get("Vary", "")
|
||||
Reference in new issue
Block a user