Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fa551de197 | ||
|
|
b2043286fa |
No files matched your search
@@ -125,6 +125,24 @@ WORKER_TIMEOUT=30
|
||||
# Max requests per worker before restart (prevents memory leaks)
|
||||
MAX_REQUESTS=1000
|
||||
|
||||
# Behind a reverse proxy (Caddy, nginx): take the client address from
|
||||
# X-Forwarded-For. Without it every visitor shares the proxy's address, so one
|
||||
# busy client can rate-limit everyone on /webmention and /websub.
|
||||
# Only enable when the proxy sets X-Forwarded-For itself (Caddy does).
|
||||
TRUSTED_PROXY=true
|
||||
|
||||
# =============================================================================
|
||||
# WEBSUB (v2.10.0)
|
||||
# =============================================================================
|
||||
|
||||
# Built-in hub at /websub for this site's feeds. Set WEBSUB_HUB_URL to use
|
||||
# another hub instead (StarPunk then pings it on changes).
|
||||
WEBSUB_ENABLED=true
|
||||
# WEBSUB_HUB_URL=https://hub.example.com/
|
||||
# WEBSUB_MAX_SUBSCRIPTIONS=1000
|
||||
# Per callback server (resolved address)
|
||||
# WEBSUB_MAX_PER_HOST=25
|
||||
|
||||
# =============================================================================
|
||||
# DEVELOPMENT OPTIONS
|
||||
# =============================================================================
|
||||
|
||||
@@ -7,6 +7,33 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [2.10.1] - 2026-11-03
|
||||
|
||||
WebSub fixes from an adversarial review of v2.10.0 against the [W3C Recommendation](https://www.w3.org/TR/websub/) (details in ADR-071, Amendments).
|
||||
|
||||
### Security
|
||||
|
||||
- One subscription request could stop the hub for good: a callback that answered verification with a large page, or a callback URL with a control character, crashed the worker on every pass. Each queued item is now handled on its own
|
||||
- A subscriber sending its response one byte at a time could hold the hub for hours. Every request the hub (and webmention fetching) makes now has a hard time limit
|
||||
- The subscription limits only counted verified subscriptions and could be dodged with subdomains. They now count pending requests too, per callback server address, and at most 200 requests can wait for verification
|
||||
- Callbacks must be printable URLs of reasonable length, not on this site, and on ports 80, 443 or 1024 and up
|
||||
|
||||
### Fixed
|
||||
|
||||
- New posts with photos were sent to subscribers (and cached in the feeds) without their photos, and photo-only edits sent the old feed. Notes with photos are announced once the photos are attached, and the feed cache follows photo and caption changes
|
||||
- A subscriber whose deliveries failed for three days was removed; WebSub says to keep it until its lease ends. Deliveries are retried and given up, the subscription stays
|
||||
- A temporary DNS failure or an error rendering the feed lost the delivery instead of retrying it. A feed that no longer exists now ends its subscriptions with a `hub.mode=denied` notice
|
||||
- Verification sent our normalised topic (`/tag/python`) instead of the one requested (`/tag/Python`), so strict subscribers refused it. Percent-encoded topics (`/%66eed.atom`) are now understood, the Content-Type check ignores case, unsubscribe requests ignore `hub.lease_seconds`, and an empty `hub.secret` is refused instead of turning signatures off
|
||||
- A newer subscription request arriving while an older one was being verified was lost (migration 017)
|
||||
- Unsubscribing works while `WEBSUB_HUB_URL` points to another hub
|
||||
- Feed ETags differed between server processes, so conditional requests rarely matched; `/feed` now sends `Vary: Accept`
|
||||
- A like no longer causes pushes of the home page and feeds, which don't show likes; each feed is rendered once per delivery round, not once per subscriber
|
||||
- Internationalised callback domains can be verified
|
||||
|
||||
### Changed
|
||||
|
||||
- `.env.example` documents `TRUSTED_PROXY` (needed behind a reverse proxy for per-client rate limits) and the WebSub settings
|
||||
|
||||
## [2.10.0] - 2026-11-03
|
||||
|
||||
### Added
|
||||
|
||||
@@ -164,3 +164,43 @@ deliveries.
|
||||
keep updated, and few maintained hubs exist.
|
||||
- **Accepting any topic** (a general-purpose hub). Not needed, and it
|
||||
would let the hub be used to fetch and push arbitrary URLs.
|
||||
|
||||
## Amendments (v2.10.1, 2026-11-03)
|
||||
|
||||
An adversarial review of v2.10.0 against the Recommendation led to these
|
||||
changes; where they differ, they replace the text above.
|
||||
|
||||
- **Worker robustness.** Each queued item is handled on its own: an
|
||||
unexpected error drops (verification) or retries (delivery) that item
|
||||
only. Before, a callback answering with a large page or containing a
|
||||
control character stopped the worker for good.
|
||||
- **Time limits.** `safe_get` and `safe_post` close the connection when
|
||||
the whole exchange exceeds its time limit (10 s), not just a single read.
|
||||
Verification replies over 4 KB fail; delivery responses are read up to
|
||||
4 KB.
|
||||
- **Leases are kept (spec 7).** A failing subscriber is not removed; its
|
||||
deliveries are retried and then given up, and the next change is
|
||||
delivered again. Only lease expiry, 410 Gone or unsubscribing end a
|
||||
subscription. DNS failures are retried. A topic that renders 404/410
|
||||
ends its subscriptions with a `hub.mode=denied` notice; other render
|
||||
errors are retried.
|
||||
- **Photos.** A note with photos is announced once they are attached
|
||||
(`attach_media_to_note` notifies), and the feed cache key and ETag
|
||||
include `content_version`, so photo and caption changes reach feeds and
|
||||
subscribers. ETags no longer depend on the feed's build time.
|
||||
- **Subscription requests (spec 5.1).** Unreserved percent-encodings are
|
||||
decoded; the Content-Type check is case-insensitive; `hub.secret` and
|
||||
`hub.lease_seconds` are ignored for unsubscriptions; an empty secret is
|
||||
refused; verification echoes `hub.topic` exactly as requested
|
||||
(`requested_topic`); unsubscribing works while an external hub is
|
||||
configured. Callbacks must be printable, at most 2,048 characters, not on
|
||||
this site, on port 80, 443 or 1024+, and are stored with an IDNA host.
|
||||
- **Limits** count pending requests as well as subscriptions, per callback
|
||||
*address* (so subdomains on one server share a limit), with at most 200
|
||||
pending requests. Rate limiting needs `TRUSTED_PROXY=true` behind Caddy.
|
||||
- **Races.** Each request has a `request_id`; verifying an older request
|
||||
doesn't consume a newer one (migration 017).
|
||||
- **Less work.** A topic is rendered once per worker pass; likes only
|
||||
update `/likes` and tag pages.
|
||||
- Corrections: there is no `state` column (a subscription row exists only
|
||||
once verified); expired leases are pruned hourly.
|
||||
@@ -0,0 +1,12 @@
|
||||
-- Migration 017: WebSub hardening (v2.10.1)
|
||||
-- request_id: each subscription request gets its own id, so verifying an
|
||||
-- older request can't consume a newer one for the same topic and callback.
|
||||
-- requested_topic: the hub.topic exactly as sent; verification echoes it.
|
||||
-- callback_ip: address the callback resolved to, for per-address limits.
|
||||
|
||||
ALTER TABLE websub_intents ADD COLUMN request_id TEXT;
|
||||
ALTER TABLE websub_intents ADD COLUMN requested_topic TEXT;
|
||||
ALTER TABLE websub_intents ADD COLUMN callback_ip TEXT;
|
||||
ALTER TABLE websub_subscriptions ADD COLUMN callback_ip TEXT;
|
||||
CREATE INDEX IF NOT EXISTS idx_websub_subscriptions_ip ON websub_subscriptions(callback_ip);
|
||||
CREATE INDEX IF NOT EXISTS idx_websub_intents_ip ON websub_intents(callback_ip);
|
||||
@@ -470,5 +470,5 @@ def create_app(config=None):
|
||||
|
||||
# Package version (Semantic Versioning 2.0.0)
|
||||
# See docs/standards/versioning-strategy.md for details
|
||||
__version__ = "2.10.0"
|
||||
__version_info__ = (2, 10, 0)
|
||||
__version__ = "2.10.1"
|
||||
__version_info__ = (2, 10, 1)
|
||||
@@ -169,7 +169,11 @@ class FeedCache:
|
||||
- Adds new entry or updates existing entry
|
||||
"""
|
||||
cache_key = self._generate_cache_key(format_name, notes_checksum)
|
||||
etag = self._generate_etag(content)
|
||||
# From what the feed is made of, not its bytes: the bytes include a
|
||||
# build time, so each server process made a different ETag and
|
||||
# conditional requests rarely matched (v2.10.1). Weak ETag: same
|
||||
# notes, semantically the same feed.
|
||||
etag = self._generate_etag(f"{format_name}:{notes_checksum}")
|
||||
timestamp = time.time()
|
||||
|
||||
# Evict if cache is full
|
||||
|
||||
@@ -891,6 +891,15 @@ def attach_media_to_note(note_id: int, media_ids: List[int], captions: List[str]
|
||||
db.rollback()
|
||||
raise
|
||||
|
||||
# Feeds and pages show the photos: refresh caches and tell WebSub (v2.10.1)
|
||||
try:
|
||||
from starpunk.notes import _websub_changed, get_note
|
||||
note = get_note(id=note_id, load_content=False)
|
||||
if note is not None:
|
||||
_websub_changed(note, was_public=note.published)
|
||||
except Exception as e:
|
||||
current_app.logger.warning(f"Failed to queue WebSub update for note {note_id}: {e}")
|
||||
|
||||
|
||||
def get_note_media(note_id: int) -> List[Dict]:
|
||||
"""
|
||||
|
||||
@@ -760,10 +760,7 @@ def handle_update(data: dict, token_info: dict):
|
||||
if "photo" in touched:
|
||||
from starpunk.media import attach_media_to_note
|
||||
attach_media_to_note(note.id, media_ids, captions)
|
||||
note_changed(note.id)
|
||||
if not changes: # update_note already told WebSub otherwise
|
||||
from starpunk import websub
|
||||
websub.note_changed(note, was_public=note.published)
|
||||
note_changed(note.id) # attach_media_to_note tells WebSub
|
||||
except InvalidNoteDataError as e:
|
||||
raise MicropubValidationError(str(e))
|
||||
except MicropubError:
|
||||
|
||||
+4
-1
@@ -374,7 +374,10 @@ def create_note(
|
||||
# Reply context fetch failure should not prevent note creation
|
||||
current_app.logger.warning(f"Failed to fetch reply context for {in_reply_to}: {e}")
|
||||
|
||||
_websub_changed(note)
|
||||
# A note with photos is announced when they're attached
|
||||
# (attach_media_to_note), so subscribers never get it without them
|
||||
if not has_media:
|
||||
_websub_changed(note)
|
||||
return note
|
||||
|
||||
|
||||
|
||||
@@ -150,13 +150,17 @@ def create_note_submit():
|
||||
# Parse tags (v1.3.0 Phase 3)
|
||||
tags = parse_tag_input(tags_input)
|
||||
|
||||
# With photos, WebSub hears about the note once they're attached (v2.10.1)
|
||||
has_uploads = any(f.filename for f in request.files.getlist('media_files'))
|
||||
|
||||
try:
|
||||
# Create note first (per Q4)
|
||||
note = create_note(
|
||||
content,
|
||||
published=published,
|
||||
custom_slug=custom_slug if custom_slug else None,
|
||||
tags=tags if tags else None
|
||||
tags=tags if tags else None,
|
||||
has_media=has_uploads,
|
||||
)
|
||||
|
||||
# Handle media uploads (v1.2.0 Phase 3)
|
||||
@@ -195,6 +199,11 @@ def create_note_submit():
|
||||
if errors:
|
||||
flash(f"Note created, but some images failed: {'; '.join(errors)}", "warning")
|
||||
|
||||
if not media_ids:
|
||||
# No photo made it; announce the note as it is (v2.10.1)
|
||||
from starpunk.notes import _websub_changed
|
||||
_websub_changed(note)
|
||||
|
||||
# Crosspost after media is attached so images go out too (v2.1.0)
|
||||
if "syndication_form" in request.form:
|
||||
from starpunk.syndication import select_targets
|
||||
|
||||
@@ -141,7 +141,12 @@ def _generate_feed_with_cache(format_name: str, non_streaming_generator):
|
||||
|
||||
# Caching enabled - use FeedCache
|
||||
feed_cache = get_cache()
|
||||
notes_checksum = feed_cache.generate_notes_checksum(notes)
|
||||
# The content version covers changes the note list doesn't show, such as
|
||||
# photos attached or captions edited (v2.10.1)
|
||||
from starpunk.database import get_db
|
||||
from starpunk.websub import get_content_version
|
||||
notes_checksum = (feed_cache.generate_notes_checksum(notes)
|
||||
+ ":" + get_content_version(get_db(current_app)))
|
||||
|
||||
# Check If-None-Match header for conditional requests
|
||||
if_none_match = request.headers.get('If-None-Match')
|
||||
@@ -508,15 +513,15 @@ def feed():
|
||||
)
|
||||
|
||||
# Route to appropriate generator
|
||||
if format_name == 'rss':
|
||||
return feed_rss()
|
||||
elif format_name == 'atom':
|
||||
return feed_atom()
|
||||
if format_name == 'atom':
|
||||
response = feed_atom()
|
||||
elif format_name == 'json':
|
||||
return feed_json()
|
||||
response = feed_json()
|
||||
else:
|
||||
# Shouldn't reach here, but be defensive
|
||||
return feed_rss()
|
||||
response = feed_rss()
|
||||
# The format (and its rel=self) depends on Accept (v2.10.1)
|
||||
response.headers.add('Vary', 'Accept')
|
||||
return response
|
||||
|
||||
|
||||
@bp.route("/feed.rss")
|
||||
|
||||
@@ -15,7 +15,10 @@ from starpunk import websub
|
||||
bp = Blueprint("websub", __name__)
|
||||
|
||||
# Subscription requests per client IP per minute
|
||||
# (per gunicorn process; needs TRUSTED_PROXY=true behind a reverse proxy,
|
||||
# or every client shares the proxy's address)
|
||||
_limiter = RateLimiter(limit=30, window_seconds=60)
|
||||
_requests = 0
|
||||
|
||||
|
||||
def _text(message: str, status: int):
|
||||
@@ -28,10 +31,15 @@ def hub():
|
||||
return _text("WebSub is not enabled on this site", 404)
|
||||
|
||||
from starpunk.routes.webmention import get_client_ip
|
||||
global _requests
|
||||
_requests += 1
|
||||
if _requests % 500 == 0:
|
||||
_limiter.cleanup() # forget idle clients so memory doesn't grow
|
||||
if not _limiter.is_allowed(get_client_ip()):
|
||||
return _text("Too many requests", 429)
|
||||
|
||||
if "application/x-www-form-urlencoded" not in request.headers.get("Content-Type", ""):
|
||||
# Media types are case-insensitive (v2.10.1)
|
||||
if request.mimetype != "application/x-www-form-urlencoded":
|
||||
return _text("Send an application/x-www-form-urlencoded body", 400)
|
||||
|
||||
try:
|
||||
|
||||
@@ -16,6 +16,7 @@ References:
|
||||
|
||||
import ipaddress
|
||||
import socket
|
||||
import threading
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional
|
||||
@@ -40,6 +41,11 @@ class SSRFBlockedError(Exception):
|
||||
pass
|
||||
|
||||
|
||||
class ResolutionError(ValueError):
|
||||
"""The host name could not be resolved (may be temporary; v2.10.1)"""
|
||||
pass
|
||||
|
||||
|
||||
class FetchTooLargeError(Exception):
|
||||
"""Raised when a response is larger than the limit"""
|
||||
pass
|
||||
@@ -132,7 +138,7 @@ def validate_url_for_fetch(url: str) -> str:
|
||||
try:
|
||||
return safe_resolve(parsed.hostname)
|
||||
except socket.gaierror as e:
|
||||
raise ValueError(f"Could not resolve {parsed.hostname}: {e}")
|
||||
raise ResolutionError(f"Could not resolve {parsed.hostname}: {e}")
|
||||
|
||||
|
||||
def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
|
||||
@@ -158,6 +164,38 @@ def _pinned_request(client: httpx.Client, method: str, url: str, headers: dict,
|
||||
)
|
||||
|
||||
|
||||
class _Watchdog:
|
||||
"""
|
||||
Close a client when its deadline passes (v2.10.1)
|
||||
|
||||
httpx timeouts apply per read, so a server sending one byte at a time
|
||||
could hold a request for hours; closing the client ends it.
|
||||
"""
|
||||
|
||||
def __init__(self, client: httpx.Client, seconds: float):
|
||||
self.fired = False
|
||||
|
||||
def fire():
|
||||
self.fired = True
|
||||
try:
|
||||
client.close()
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
self._timer = threading.Timer(max(seconds, 0.01), fire)
|
||||
self._timer.daemon = True
|
||||
|
||||
def __enter__(self):
|
||||
self._timer.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, exc_type, exc, tb):
|
||||
self._timer.cancel()
|
||||
if self.fired and exc_type is not None and not isinstance(exc, httpx.TimeoutException):
|
||||
raise httpx.TimeoutException("Exceeded time limit") from exc
|
||||
return False
|
||||
|
||||
|
||||
@dataclass
|
||||
class FetchResult:
|
||||
"""A fetched response, after redirects"""
|
||||
@@ -177,28 +215,34 @@ class FetchResult:
|
||||
|
||||
def safe_post(url: str, data: Optional[dict] = None, headers: Optional[dict] = None,
|
||||
timeout: float = TIMEOUT, content: Optional[bytes] = None,
|
||||
max_bytes: int = 64 * 1024) -> FetchResult:
|
||||
max_bytes: int = 4096) -> FetchResult:
|
||||
"""
|
||||
POST form data (or a raw body) to a checked, pinned address.
|
||||
|
||||
Redirects are not followed. Only the first max_bytes of the response are
|
||||
read (callers need the status and maybe an error message).
|
||||
Redirects are not followed. `timeout` bounds the whole exchange, so a
|
||||
server trickling its response can't hold the caller (v2.10.1). Only the
|
||||
first max_bytes of the response are read (callers need the status and
|
||||
maybe an error message).
|
||||
|
||||
Raises:
|
||||
SSRFBlockedError, ValueError: URL is not allowed
|
||||
httpx.HTTPError: Network errors
|
||||
"""
|
||||
kwargs = {'content': content} if content is not None else {'data': data}
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client:
|
||||
deadline = time.monotonic() + timeout
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT, timeout=timeout) as client, \
|
||||
_Watchdog(client, timeout):
|
||||
request = _pinned_request(client, 'POST', url, headers or {}, **kwargs)
|
||||
response = client.send(request, stream=True)
|
||||
try:
|
||||
body = b''
|
||||
for chunk in response.iter_bytes(chunk_size=8192):
|
||||
for chunk in response.iter_bytes(chunk_size=1024):
|
||||
body += chunk
|
||||
if len(body) >= max_bytes:
|
||||
body = body[:max_bytes]
|
||||
break
|
||||
if time.monotonic() > deadline:
|
||||
raise httpx.TimeoutException("Response exceeded time limit")
|
||||
return FetchResult(url=url, status_code=response.status_code, headers=response.headers,
|
||||
content=body, encoding=response.charset_encoding or 'utf-8')
|
||||
finally:
|
||||
@@ -227,7 +271,8 @@ def safe_get(url: str, headers: dict, max_bytes: int = MAX_BYTES,
|
||||
"""
|
||||
deadline = time.monotonic() + timeout
|
||||
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client:
|
||||
with httpx.Client(follow_redirects=False, transport=TRANSPORT) as client, \
|
||||
_Watchdog(client, timeout):
|
||||
for _ in range(max_redirects + 1):
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
|
||||
+152
-49
@@ -97,17 +97,44 @@ def topic_for_request(response=None) -> Optional[str]:
|
||||
return None
|
||||
|
||||
|
||||
_UNRESERVED = set("ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-._~")
|
||||
|
||||
|
||||
def decode_unreserved(value: str) -> str:
|
||||
"""
|
||||
Decode percent-encoded unreserved characters (%66 -> f), leaving the rest
|
||||
|
||||
Spec 5.1.1: hubs MUST always decode non-reserved characters in URL
|
||||
parameters, so /%66eed.atom and /feed.atom are the same topic.
|
||||
"""
|
||||
out, i = [], 0
|
||||
while i < len(value):
|
||||
if value[i] == "%" and i + 3 <= len(value):
|
||||
try:
|
||||
char = chr(int(value[i + 1:i + 3], 16))
|
||||
except ValueError:
|
||||
char = None
|
||||
if char is not None and char in _UNRESERVED:
|
||||
out.append(char)
|
||||
i += 3
|
||||
continue
|
||||
out.append(value[i])
|
||||
i += 1
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def canonical_topic(url: str) -> Optional[str]:
|
||||
"""
|
||||
A subscription's hub.topic as one of this site's topics, or None
|
||||
|
||||
Tag topics must name an existing tag.
|
||||
Unreserved percent-encodings are decoded and the scheme and host are
|
||||
compared case-insensitively. Tag topics must name an existing tag.
|
||||
"""
|
||||
if not url:
|
||||
return None
|
||||
site = urlparse(site_url())
|
||||
parsed = urlparse(url.strip())
|
||||
if (parsed.scheme, parsed.netloc.lower()) != (site.scheme, site.netloc.lower()):
|
||||
parsed = urlparse(decode_unreserved(url.strip()))
|
||||
if (parsed.scheme.lower(), parsed.netloc.lower()) != (site.scheme, site.netloc.lower()):
|
||||
return None
|
||||
path = parsed.path or "/"
|
||||
if parsed.query or parsed.fragment:
|
||||
@@ -115,8 +142,9 @@ def canonical_topic(url: str) -> Optional[str]:
|
||||
if path in set(PAGE_TOPICS.values()) | set(FEED_TOPICS.values()):
|
||||
return topic_url(path)
|
||||
if path.startswith("/tag/"):
|
||||
from urllib.parse import unquote
|
||||
from starpunk.tags import get_tag_by_name, normalize_tag
|
||||
name = normalize_tag(path[len("/tag/"):])[0]
|
||||
name = normalize_tag(unquote(path[len("/tag/"):]))[0]
|
||||
if name and get_tag_by_name(name):
|
||||
return tag_topic(name)
|
||||
return None
|
||||
@@ -214,9 +242,11 @@ def note_changed(note, was_public: bool = False, old_tags: Iterable[str] = ()) -
|
||||
tags = set()
|
||||
tags |= set(old_tags)
|
||||
|
||||
topics = [topic_url(p) for p in ("/", "/feed.atom", "/feed.rss", "/feed.json")]
|
||||
# Likes appear on /likes, not on the home page or in the feeds
|
||||
if getattr(note, "like_of", None):
|
||||
topics.append(topic_url("/likes"))
|
||||
topics = [topic_url("/likes")]
|
||||
else:
|
||||
topics = [topic_url(p) for p in ("/", "/feed.atom", "/feed.rss", "/feed.json")]
|
||||
topics += [tag_topic(name) for name in sorted(tags)]
|
||||
queue_topics(db, topics)
|
||||
|
||||
@@ -249,6 +279,50 @@ def queue_topics(db, topics: Iterable[str]) -> int:
|
||||
|
||||
# --- Hub: subscription requests (spec 5.1) -------------------------------------
|
||||
|
||||
MAX_CALLBACK_LENGTH = 2048
|
||||
MAX_PENDING_INTENTS = 200
|
||||
ALLOWED_PORTS = {80, 443} # plus any port from 1024 up
|
||||
|
||||
|
||||
def _clean_callback(callback: str) -> str:
|
||||
"""
|
||||
Check a hub.callback and return it with an ASCII (IDNA) host
|
||||
|
||||
Raises:
|
||||
SubscriptionError: if it is not an acceptable callback URL
|
||||
"""
|
||||
import httpx
|
||||
|
||||
if not callback or len(callback) > MAX_CALLBACK_LENGTH:
|
||||
raise SubscriptionError(f"hub.callback must be a URL of at most {MAX_CALLBACK_LENGTH} characters")
|
||||
if any(ord(c) < 0x20 or ord(c) == 0x7f for c in callback) or " " in callback:
|
||||
raise SubscriptionError("hub.callback contains spaces or control characters")
|
||||
parsed = urlparse(callback)
|
||||
if parsed.scheme.lower() not in ("http", "https") or not parsed.hostname:
|
||||
raise SubscriptionError("hub.callback must be an absolute http(s) URL")
|
||||
try:
|
||||
port = parsed.port
|
||||
except ValueError:
|
||||
raise SubscriptionError("hub.callback has an invalid port")
|
||||
if port is not None and port not in ALLOWED_PORTS and port < 1024:
|
||||
raise SubscriptionError("hub.callback may not use that port")
|
||||
|
||||
try:
|
||||
host = parsed.hostname.encode("idna").decode("ascii")
|
||||
except UnicodeError:
|
||||
raise SubscriptionError("hub.callback has an invalid host name")
|
||||
netloc = host + (f":{port}" if port is not None else "")
|
||||
callback = parsed._replace(scheme=parsed.scheme.lower(), netloc=netloc).geturl()
|
||||
|
||||
if host.lower() == (urlparse(site_url()).hostname or "").lower():
|
||||
raise SubscriptionError("hub.callback may not be on this site")
|
||||
try:
|
||||
httpx.URL(callback)
|
||||
except Exception:
|
||||
raise SubscriptionError("hub.callback is not a valid URL")
|
||||
return callback
|
||||
|
||||
|
||||
def accept_subscription_request(form, db) -> None:
|
||||
"""
|
||||
Validate a subscription request and queue it for verification
|
||||
@@ -258,71 +332,100 @@ def accept_subscription_request(form, db) -> None:
|
||||
"""
|
||||
from starpunk.webmention_ssrf import SSRFBlockedError, validate_url_for_fetch
|
||||
|
||||
if external_hub():
|
||||
raise SubscriptionError("This site uses another hub: " + external_hub(), 400)
|
||||
|
||||
mode = (form.get("hub.mode") or "").strip()
|
||||
if mode not in ("subscribe", "unsubscribe"):
|
||||
raise SubscriptionError("hub.mode must be subscribe or unsubscribe")
|
||||
if mode == "subscribe" and external_hub():
|
||||
# Unsubscribes are still taken, so old subscribers can leave
|
||||
raise SubscriptionError("This site uses another hub: " + external_hub(), 400)
|
||||
|
||||
callback = (form.get("hub.callback") or "").strip()
|
||||
parsed = urlparse(callback)
|
||||
if parsed.scheme not in ("http", "https") or not parsed.hostname:
|
||||
raise SubscriptionError("hub.callback must be an absolute http(s) URL")
|
||||
callback = _clean_callback((form.get("hub.callback") or "").strip())
|
||||
try:
|
||||
validate_url_for_fetch(callback)
|
||||
callback_ip = validate_url_for_fetch(callback)
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
raise SubscriptionError(f"hub.callback is not reachable: {e}")
|
||||
|
||||
topic = canonical_topic(form.get("hub.topic") or "")
|
||||
requested_topic = (form.get("hub.topic") or "").strip()
|
||||
topic = canonical_topic(requested_topic)
|
||||
if not topic:
|
||||
raise SubscriptionError("hub.topic is not a topic of this site; use the rel=self URL")
|
||||
|
||||
secret = form.get("hub.secret")
|
||||
if mode == "unsubscribe":
|
||||
secret = None # MUST be ignored for unsubscriptions
|
||||
if secret is not None and len(secret.encode("utf-8")) >= MAX_SECRET_BYTES:
|
||||
raise SubscriptionError("hub.secret must be less than 200 bytes")
|
||||
|
||||
lease = LEASE_DEFAULT
|
||||
if form.get("hub.lease_seconds"):
|
||||
try:
|
||||
lease = int(form["hub.lease_seconds"])
|
||||
except ValueError:
|
||||
raise SubscriptionError("hub.lease_seconds must be a number")
|
||||
lease = max(LEASE_MIN, min(lease, LEASE_MAX))
|
||||
|
||||
secret, lease = None, None
|
||||
if mode == "subscribe":
|
||||
existing = db.execute(
|
||||
"SELECT 1 FROM websub_subscriptions WHERE topic = ? AND callback = ?", (topic, callback)
|
||||
).fetchone()
|
||||
if not existing:
|
||||
total = db.execute("SELECT COUNT(*) FROM websub_subscriptions").fetchone()[0]
|
||||
if total >= current_app.config.get("WEBSUB_MAX_SUBSCRIPTIONS", 1000):
|
||||
raise SubscriptionError("This hub has reached its subscription limit", 429)
|
||||
host = parsed.hostname.lower()
|
||||
per_host = sum(
|
||||
1 for (cb,) in db.execute("SELECT callback FROM websub_subscriptions")
|
||||
if (urlparse(cb).hostname or "").lower() == host
|
||||
)
|
||||
if per_host >= current_app.config.get("WEBSUB_MAX_PER_HOST", 25):
|
||||
raise SubscriptionError(f"Too many subscriptions for {host}", 429)
|
||||
# hub.secret and hub.lease_seconds MUST be ignored for unsubscriptions
|
||||
secret = form.get("hub.secret")
|
||||
if secret is not None:
|
||||
if secret == "":
|
||||
raise SubscriptionError("hub.secret may not be empty")
|
||||
if len(secret.encode("utf-8")) >= MAX_SECRET_BYTES:
|
||||
raise SubscriptionError("hub.secret must be less than 200 bytes")
|
||||
lease = LEASE_DEFAULT
|
||||
if form.get("hub.lease_seconds"):
|
||||
try:
|
||||
lease = int(form["hub.lease_seconds"])
|
||||
except ValueError:
|
||||
raise SubscriptionError("hub.lease_seconds must be a number")
|
||||
lease = max(LEASE_MIN, min(lease, LEASE_MAX))
|
||||
_check_limits(db, topic, callback, callback_ip)
|
||||
|
||||
# The latest request for a topic and callback replaces any pending one
|
||||
# Each request gets its own id; the latest replaces any pending one
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO websub_intents (topic, callback, mode, secret, lease_seconds)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
INSERT INTO websub_intents
|
||||
(topic, callback, mode, secret, lease_seconds, request_id, requested_topic, callback_ip)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(topic, callback) DO UPDATE SET
|
||||
mode = excluded.mode, secret = excluded.secret,
|
||||
lease_seconds = excluded.lease_seconds, attempts = 0,
|
||||
next_attempt_at = CURRENT_TIMESTAMP, created_at = CURRENT_TIMESTAMP
|
||||
lease_seconds = excluded.lease_seconds, request_id = excluded.request_id,
|
||||
requested_topic = excluded.requested_topic, callback_ip = excluded.callback_ip,
|
||||
attempts = 0, next_attempt_at = CURRENT_TIMESTAMP, created_at = CURRENT_TIMESTAMP
|
||||
""",
|
||||
(topic, callback, mode, secret, lease),
|
||||
(topic, callback, mode, secret, lease, secrets.token_hex(16), requested_topic, callback_ip),
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _check_limits(db, topic: str, callback: str, callback_ip: str) -> None:
|
||||
"""
|
||||
Subscription limits, counting pending requests too (ADR-071 4)
|
||||
|
||||
Per callback address rather than host name, so many subdomains on one
|
||||
server share a limit. Renewing an existing subscription is always fine.
|
||||
"""
|
||||
if db.execute("SELECT 1 FROM websub_subscriptions WHERE topic = ? AND callback = ?",
|
||||
(topic, callback)).fetchone():
|
||||
return
|
||||
|
||||
pending = db.execute("SELECT COUNT(*) FROM websub_intents").fetchone()[0]
|
||||
if pending >= MAX_PENDING_INTENTS:
|
||||
raise SubscriptionError("Too many subscription requests waiting; try again later", 429)
|
||||
|
||||
total = db.execute(
|
||||
"""
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT topic, callback FROM websub_subscriptions
|
||||
UNION
|
||||
SELECT topic, callback FROM websub_intents WHERE mode = 'subscribe'
|
||||
)
|
||||
"""
|
||||
).fetchone()[0]
|
||||
if total >= current_app.config.get("WEBSUB_MAX_SUBSCRIPTIONS", 1000):
|
||||
raise SubscriptionError("This hub has reached its subscription limit", 429)
|
||||
|
||||
per_address = db.execute(
|
||||
"""
|
||||
SELECT COUNT(*) FROM (
|
||||
SELECT topic, callback FROM websub_subscriptions WHERE callback_ip = ?
|
||||
UNION
|
||||
SELECT topic, callback FROM websub_intents WHERE mode = 'subscribe' AND callback_ip = ?
|
||||
)
|
||||
""",
|
||||
(callback_ip, callback_ip),
|
||||
).fetchone()[0]
|
||||
if per_address >= current_app.config.get("WEBSUB_MAX_PER_HOST", 25):
|
||||
raise SubscriptionError("Too many subscriptions for that callback's server", 429)
|
||||
|
||||
|
||||
def stats(db) -> dict:
|
||||
"""Active subscribers per topic and deliveries being retried (admin)"""
|
||||
by_topic = db.execute(
|
||||
|
||||
+152
-77
@@ -4,13 +4,17 @@ WebSub worker: verification of intent and content distribution (v2.10.0)
|
||||
Runs like the Webmention worker: a daemon thread in every gunicorn process,
|
||||
but only the process holding the file lock works the queues.
|
||||
|
||||
- Intents (spec 5.3): GET the callback with a challenge; 2xx echoing it
|
||||
- Intents (spec 5.3): GET the callback with a challenge; a 2xx echoing it
|
||||
verifies. Anything else (including redirects) leaves the subscription as
|
||||
it was. Network errors and 5xx are retried a few times.
|
||||
- Deliveries (spec 6, 7): render the topic in-app and POST it with
|
||||
it was. Network errors, DNS failures and 5xx are retried a few times.
|
||||
- Deliveries (spec 7): render the topic in-app and POST it with
|
||||
Link rel=hub/self and, if there is a secret, X-Hub-Signature (sha256).
|
||||
Failures are retried; the subscription stays until its lease ends.
|
||||
- External hub: a delivery without a subscription is a publish ping.
|
||||
|
||||
v2.10.1: every queue item is handled on its own, so one bad item can't stop
|
||||
the worker; all outbound requests have a hard time limit.
|
||||
|
||||
References:
|
||||
- https://www.w3.org/TR/websub/
|
||||
- ADR-071
|
||||
@@ -32,9 +36,9 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
DELIVERY_BACKOFF = [60, 300, 1800, 7200, 21600] # then give up on this delivery
|
||||
INTENT_BACKOFF = [60, 300, 900]
|
||||
DROP_AFTER = timedelta(days=3) # failing this long without a success: unsubscribe
|
||||
PRUNE_EVERY = timedelta(hours=1)
|
||||
BATCH = 10
|
||||
VERIFY_MAX_BYTES = 4096 # a challenge reply is short; larger means not verified
|
||||
|
||||
|
||||
def callback_with_params(callback: str, params: dict) -> str:
|
||||
@@ -114,78 +118,118 @@ class WebSubWorker(threading.Thread):
|
||||
|
||||
db = get_db(self.app)
|
||||
done = 0
|
||||
rendered = {} # topic -> (status, content type, body), once per pass
|
||||
|
||||
for row in db.execute(
|
||||
"SELECT * FROM websub_intents WHERE next_attempt_at <= ? ORDER BY id LIMIT ?",
|
||||
"SELECT * FROM websub_intents WHERE next_attempt_at <= ? ORDER BY next_attempt_at, id LIMIT ?",
|
||||
(_now(), BATCH),
|
||||
).fetchall():
|
||||
self.verify(db, row)
|
||||
self._guarded(db, row, "verification", self.verify, self._intent_crashed)
|
||||
done += 1
|
||||
for row in db.execute(
|
||||
"SELECT * FROM websub_deliveries WHERE next_attempt_at <= ? ORDER BY id LIMIT ?",
|
||||
"SELECT * FROM websub_deliveries WHERE next_attempt_at <= ? ORDER BY next_attempt_at, id LIMIT ?",
|
||||
(_now(), BATCH),
|
||||
).fetchall():
|
||||
self.deliver(db, row)
|
||||
self._guarded(db, row, "delivery", lambda d, r: self.deliver(d, r, rendered), self._retry_or_drop)
|
||||
done += 1
|
||||
if self._last_prune is None or datetime.utcnow() - self._last_prune > PRUNE_EVERY:
|
||||
self.prune(db)
|
||||
return done
|
||||
|
||||
def _guarded(self, db, row, what, handler, on_crash) -> None:
|
||||
"""Handle one queue item; an unexpected error affects only that item"""
|
||||
try:
|
||||
handler(db, row)
|
||||
except Exception as e:
|
||||
logger.error(f"WebSub {what} {row['id']} failed unexpectedly: {e}", exc_info=True)
|
||||
try:
|
||||
db.rollback()
|
||||
on_crash(db, row, f"{type(e).__name__}: {e}"[:200])
|
||||
db.commit()
|
||||
except Exception as cleanup:
|
||||
logger.error(f"WebSub could not record the failure: {cleanup}")
|
||||
|
||||
def _intent_crashed(self, db, intent, error) -> None:
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
|
||||
(intent["id"], intent["request_id"]))
|
||||
|
||||
# --- verification of intent (spec 5.3) ----------------------------------
|
||||
|
||||
def verify(self, db, intent) -> None:
|
||||
from starpunk.websub import new_challenge
|
||||
from starpunk.webmention_ssrf import SSRFBlockedError, safe_get, user_agent
|
||||
from starpunk.webmention_ssrf import (
|
||||
FetchTooLargeError, ResolutionError, SSRFBlockedError, safe_get, user_agent,
|
||||
)
|
||||
|
||||
challenge = new_challenge()
|
||||
params = {"hub.mode": intent["mode"], "hub.topic": intent["topic"], "hub.challenge": challenge}
|
||||
# The topic exactly as the subscriber sent it (spec 5.3)
|
||||
params = {"hub.mode": intent["mode"],
|
||||
"hub.topic": intent["requested_topic"] or intent["topic"],
|
||||
"hub.challenge": challenge}
|
||||
if intent["mode"] == "subscribe":
|
||||
params["hub.lease_seconds"] = intent["lease_seconds"]
|
||||
url = callback_with_params(intent["callback"], params)
|
||||
|
||||
retry = False
|
||||
try:
|
||||
result = safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
|
||||
max_redirects=0, max_bytes=64 * 1024)
|
||||
verified = 200 <= result.status_code < 300 and result.text.strip() == challenge
|
||||
max_redirects=0, max_bytes=VERIFY_MAX_BYTES)
|
||||
# Exact match; a trailing newline from a framework is tolerated
|
||||
verified = 200 <= result.status_code < 300 and result.text.rstrip("\r\n") == challenge
|
||||
retry = result.status_code >= 500
|
||||
error = None if verified else f"HTTP {result.status_code}" + (
|
||||
"" if result.status_code >= 300 else ": challenge not echoed")
|
||||
except ResolutionError as e:
|
||||
verified, retry, error = False, True, str(e)
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
verified, retry, error = False, False, f"Blocked: {e}"
|
||||
verified, error = False, f"Blocked: {e}"
|
||||
except FetchTooLargeError:
|
||||
verified, error = False, "Reply too large to be a challenge"
|
||||
except httpx.TooManyRedirects:
|
||||
verified, retry, error = False, False, "Redirected (not allowed)"
|
||||
verified, error = False, "Redirected (not allowed)"
|
||||
except httpx.HTTPError as e:
|
||||
verified, retry, error = False, True, str(e)[:200]
|
||||
verified, retry, error = False, True, (str(e) or type(e).__name__)[:200]
|
||||
|
||||
# Only act if no newer request replaced this one meanwhile
|
||||
current = db.execute("SELECT request_id FROM websub_intents WHERE id = ?", (intent["id"],)).fetchone()
|
||||
if current is None or current[0] != intent["request_id"]:
|
||||
return
|
||||
|
||||
if verified:
|
||||
if intent["mode"] == "subscribe":
|
||||
db.execute(
|
||||
"""
|
||||
INSERT INTO websub_subscriptions
|
||||
(topic, callback, secret, lease_seconds, expires_at, verified_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
(topic, callback, secret, lease_seconds, expires_at, verified_at, callback_ip)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(topic, callback) DO UPDATE SET
|
||||
secret = excluded.secret, lease_seconds = excluded.lease_seconds,
|
||||
expires_at = excluded.expires_at, verified_at = excluded.verified_at,
|
||||
last_error = NULL
|
||||
callback_ip = excluded.callback_ip, last_error = NULL
|
||||
""",
|
||||
(intent["topic"], intent["callback"], intent["secret"], intent["lease_seconds"],
|
||||
_later(intent["lease_seconds"]), _now()),
|
||||
_later(intent["lease_seconds"]), _now(), intent["callback_ip"]),
|
||||
)
|
||||
else:
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE topic = ? AND callback = ?",
|
||||
(intent["topic"], intent["callback"]))
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ?", (intent["id"],))
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
|
||||
(intent["id"], intent["request_id"]))
|
||||
logger.info(f"WebSub {intent['mode']} verified: {intent['callback']} -> {intent['topic']}")
|
||||
elif retry and intent["attempts"] < len(INTENT_BACKOFF):
|
||||
db.execute("UPDATE websub_intents SET attempts = attempts + 1, next_attempt_at = ? WHERE id = ?",
|
||||
(_later(INTENT_BACKOFF[intent["attempts"]]), intent["id"]))
|
||||
db.execute(
|
||||
"UPDATE websub_intents SET attempts = attempts + 1, next_attempt_at = ? "
|
||||
"WHERE id = ? AND request_id IS ?",
|
||||
(_later(INTENT_BACKOFF[intent["attempts"]]), intent["id"], intent["request_id"]),
|
||||
)
|
||||
else:
|
||||
# Verification failed: the subscription state stays as it was
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ?", (intent["id"],))
|
||||
db.execute("DELETE FROM websub_intents WHERE id = ? AND request_id IS ?",
|
||||
(intent["id"], intent["request_id"]))
|
||||
logger.info(f"WebSub {intent['mode']} not verified for {intent['callback']}: {error}")
|
||||
db.commit()
|
||||
|
||||
# --- content distribution (spec 6, 7) ------------------------------------
|
||||
# --- content distribution (spec 7) ---------------------------------------
|
||||
|
||||
def render_topic(self, topic: str):
|
||||
"""(status, content_type, body) of a topic, rendered in-app"""
|
||||
@@ -193,85 +237,116 @@ class WebSubWorker(threading.Thread):
|
||||
response = self.app.test_client().get(path, headers={"Accept": "*/*"})
|
||||
return response.status_code, response.headers.get("Content-Type", ""), response.get_data()
|
||||
|
||||
def deliver(self, db, delivery) -> None:
|
||||
def deliver(self, db, delivery, rendered: Optional[dict] = None) -> None:
|
||||
from starpunk.websub import hub_url
|
||||
from starpunk.webmention_ssrf import SSRFBlockedError, safe_post, user_agent
|
||||
from starpunk.webmention_ssrf import ResolutionError, SSRFBlockedError, safe_post, user_agent
|
||||
|
||||
rendered = {} if rendered is None else rendered
|
||||
site = self.app.config["SITE_URL"]
|
||||
sub = None
|
||||
try:
|
||||
if delivery["subscription_id"] is None:
|
||||
# External hub: ping it (hub.url is the de facto parameter)
|
||||
result = safe_post(hub_url(), data={"hub.mode": "publish", "hub.url": delivery["topic"],
|
||||
"hub.topic": delivery["topic"]},
|
||||
headers={"User-Agent": user_agent(site)})
|
||||
else:
|
||||
sub = db.execute(
|
||||
"SELECT * FROM websub_subscriptions WHERE id = ? AND expires_at > ?",
|
||||
(delivery["subscription_id"], _now()),
|
||||
).fetchone()
|
||||
if sub is None: # unsubscribed or expired meanwhile
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
db.commit()
|
||||
return
|
||||
status, content_type, body = self.render_topic(sub["topic"])
|
||||
if status != 200: # e.g. a tag page whose last note went
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
db.commit()
|
||||
return
|
||||
headers = {
|
||||
"Content-Type": content_type,
|
||||
"Link": f'<{hub_url()}>; rel="hub", <{sub["topic"]}>; rel="self"',
|
||||
"User-Agent": user_agent(site),
|
||||
}
|
||||
if sub["secret"]:
|
||||
headers["X-Hub-Signature"] = signature(sub["secret"], body)
|
||||
result = safe_post(sub["callback"], content=body, headers=headers)
|
||||
status, error = result.status_code, f"HTTP {result.status_code}"
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
status, error = None, f"Blocked: {e}"
|
||||
except httpx.HTTPError as e:
|
||||
status, error = None, str(e)[:200] or type(e).__name__
|
||||
if delivery["subscription_id"] is None:
|
||||
# External hub: ping it (hub.url is the de facto parameter)
|
||||
target, kwargs = hub_url(), {
|
||||
"data": {"hub.mode": "publish", "hub.url": delivery["topic"], "hub.topic": delivery["topic"]},
|
||||
"headers": {"User-Agent": user_agent(site)},
|
||||
}
|
||||
else:
|
||||
sub = db.execute(
|
||||
"SELECT * FROM websub_subscriptions WHERE id = ? AND expires_at > ?",
|
||||
(delivery["subscription_id"], _now()),
|
||||
).fetchone()
|
||||
if sub is None: # unsubscribed or expired meanwhile
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
db.commit()
|
||||
return
|
||||
if sub["topic"] not in rendered:
|
||||
rendered[sub["topic"]] = self.render_topic(sub["topic"])
|
||||
status, content_type, body = rendered[sub["topic"]]
|
||||
if status in (404, 410):
|
||||
# The topic is gone: end the subscription and say so (spec 5.2)
|
||||
self._deny(sub, "The topic no longer exists")
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
db.commit()
|
||||
return
|
||||
if status != 200:
|
||||
self._retry_or_drop(db, delivery, f"Topic rendered HTTP {status}")
|
||||
db.commit()
|
||||
return
|
||||
headers = {
|
||||
"Content-Type": content_type,
|
||||
"Link": f'<{hub_url()}>; rel="hub", <{sub["topic"]}>; rel="self"',
|
||||
"User-Agent": user_agent(site),
|
||||
}
|
||||
if sub["secret"] is not None:
|
||||
headers["X-Hub-Signature"] = signature(sub["secret"], body)
|
||||
target, kwargs = sub["callback"], {"content": body, "headers": headers}
|
||||
|
||||
if status is not None and 200 <= status < 300:
|
||||
permanent = False
|
||||
try:
|
||||
result = safe_post(target, **kwargs)
|
||||
status, error = result.status_code, f"HTTP {result.status_code}"
|
||||
except ResolutionError as e:
|
||||
status, error = None, str(e) # DNS can recover: retry
|
||||
except (SSRFBlockedError, ValueError) as e:
|
||||
status, error, permanent = None, f"Blocked: {e}", True
|
||||
except httpx.HTTPError as e:
|
||||
status, error = None, (str(e) or type(e).__name__)[:200]
|
||||
|
||||
succeeded = status is not None and 200 <= status < 300
|
||||
if succeeded:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
if sub is not None:
|
||||
db.execute("UPDATE websub_subscriptions SET last_success_at = ?, last_error = NULL WHERE id = ?",
|
||||
(_now(), sub["id"]))
|
||||
elif sub is not None and status == 410:
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
logger.info(f"WebSub subscriber gone (410), removed: {sub['callback']}")
|
||||
elif delivery["attempts"] < len(DELIVERY_BACKOFF) and not error.startswith("Blocked"):
|
||||
else:
|
||||
if permanent:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
logger.warning(f"WebSub delivery of {delivery['topic']} refused: {error}")
|
||||
else:
|
||||
self._retry_or_drop(db, delivery, error)
|
||||
if sub is not None:
|
||||
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
|
||||
db.commit()
|
||||
|
||||
def _retry_or_drop(self, db, delivery, error: str) -> None:
|
||||
"""
|
||||
Retry a failed delivery with backoff, then give up on it
|
||||
|
||||
The subscription itself stays active until its lease ends (spec 7);
|
||||
the next change is delivered again.
|
||||
"""
|
||||
if delivery["attempts"] < len(DELIVERY_BACKOFF):
|
||||
db.execute(
|
||||
"UPDATE websub_deliveries SET attempts = attempts + 1, next_attempt_at = ?, last_error = ? WHERE id = ?",
|
||||
(_later(DELIVERY_BACKOFF[delivery["attempts"]]), error, delivery["id"]),
|
||||
)
|
||||
if sub is not None:
|
||||
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
|
||||
else:
|
||||
db.execute("DELETE FROM websub_deliveries WHERE id = ?", (delivery["id"],))
|
||||
logger.warning(f"WebSub delivery of {delivery['topic']} gave up: {error}")
|
||||
if sub is not None:
|
||||
self._drop_if_failing(db, sub, error)
|
||||
db.commit()
|
||||
|
||||
def _drop_if_failing(self, db, sub, error: str) -> None:
|
||||
last_ok = sub["last_success_at"] or sub["verified_at"] or sub["created_at"]
|
||||
def _deny(self, sub, reason: str) -> None:
|
||||
"""Tell a subscriber its subscription has ended (hub.mode=denied), best effort"""
|
||||
from starpunk.webmention_ssrf import safe_get, user_agent
|
||||
url = callback_with_params(sub["callback"], {"hub.mode": "denied", "hub.topic": sub["topic"],
|
||||
"hub.reason": reason})
|
||||
try:
|
||||
since = datetime.fromisoformat(str(last_ok))
|
||||
except ValueError:
|
||||
return
|
||||
if datetime.utcnow() - since > DROP_AFTER:
|
||||
db.execute("DELETE FROM websub_subscriptions WHERE id = ?", (sub["id"],))
|
||||
logger.info(f"WebSub subscription failing for {DROP_AFTER.days} days, removed: {sub['callback']}")
|
||||
else:
|
||||
db.execute("UPDATE websub_subscriptions SET last_error = ? WHERE id = ?", (error, sub["id"]))
|
||||
safe_get(url, headers={"User-Agent": user_agent(self.app.config["SITE_URL"])},
|
||||
max_redirects=0, max_bytes=VERIFY_MAX_BYTES)
|
||||
except Exception as e:
|
||||
logger.info(f"WebSub denial to {sub['callback']} not delivered: {e}")
|
||||
|
||||
# --- housekeeping -------------------------------------------------------
|
||||
|
||||
def prune(self, db) -> int:
|
||||
"""Remove subscriptions whose lease has run out (they weren't renewed)"""
|
||||
"""Remove subscriptions whose lease has run out, and their deliveries"""
|
||||
cursor = db.execute("DELETE FROM websub_subscriptions WHERE expires_at <= ?", (_now(),))
|
||||
db.execute("DELETE FROM websub_deliveries WHERE subscription_id IS NOT NULL AND "
|
||||
"subscription_id NOT IN (SELECT id FROM websub_subscriptions)")
|
||||
db.commit()
|
||||
self._last_prune = datetime.utcnow()
|
||||
if cursor.rowcount:
|
||||
|
||||
@@ -26,6 +26,9 @@ PUBLIC = [(2, 1, 6, "", ("93.184.216.34", 0))]
|
||||
def web(monkeypatch):
|
||||
"""Subscriber callbacks: handler(request) -> response, by host"""
|
||||
hosts, seen = {}, []
|
||||
# Every test client shares one IP; don't let earlier tests rate-limit this one
|
||||
from starpunk.rate_limiter import RateLimiter
|
||||
monkeypatch.setattr("starpunk.routes.websub._limiter", RateLimiter(limit=1000, window_seconds=60))
|
||||
|
||||
def handler(request):
|
||||
host = request.headers["host"]
|
||||
|
||||
@@ -0,0 +1,316 @@
|
||||
"""
|
||||
WebSub hardening (v2.10.1): one test per finding of the v2.10.0 review
|
||||
"""
|
||||
|
||||
import io
|
||||
import socket
|
||||
import sqlite3
|
||||
import threading
|
||||
import time
|
||||
from unittest.mock import patch
|
||||
from urllib.parse import parse_qs, urlparse
|
||||
|
||||
import httpx
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from tests.test_websub import ( # noqa: F401
|
||||
HUB, PUBLIC, SITE, db, echo, make_note, run_worker, subscribe, web,
|
||||
)
|
||||
from starpunk.websub_worker import WebSubWorker
|
||||
|
||||
|
||||
def jpeg(color="red"):
|
||||
buf = io.BytesIO()
|
||||
Image.new("RGB", (40, 30), color).save(buf, format="JPEG")
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def intents(app):
|
||||
return db(app).execute("SELECT * FROM websub_intents ORDER BY id").fetchall()
|
||||
|
||||
|
||||
def subs(app):
|
||||
return db(app).execute("SELECT * FROM websub_subscriptions ORDER BY id").fetchall()
|
||||
|
||||
|
||||
def subscribed(app, client, web, topic=f"{SITE}/feed.atom", callback="https://sub.example/cb"):
|
||||
hosts, seen = web
|
||||
hosts.setdefault(urlparse(callback).hostname, echo)
|
||||
subscribe(client, topic, callback=callback)
|
||||
run_worker(app)
|
||||
seen.clear()
|
||||
return hosts, seen
|
||||
|
||||
|
||||
# H1, H2: one bad item can't stop the worker -----------------------------------
|
||||
|
||||
def test_oversized_challenge_reply_does_not_wedge_worker(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["big.example"] = lambda r: httpx.Response(200, content=b"x" * 100_000)
|
||||
hosts["good.example"] = echo
|
||||
subscribe(client, f"{SITE}/", callback="https://big.example/cb")
|
||||
subscribe(client, f"{SITE}/", callback="https://good.example/cb")
|
||||
run_worker(app)
|
||||
assert [s["callback"] for s in subs(app)] == ["https://good.example/cb"]
|
||||
assert intents(app) == []
|
||||
|
||||
|
||||
@pytest.mark.parametrize("callback", [
|
||||
"https://example.com/\x00", "https://example.com/a b", "https://example.com:25/cb",
|
||||
f"{SITE}/cb", "https://" + "a" * 2100 + ".example/",
|
||||
])
|
||||
def test_bad_callbacks_refused(client, web, callback):
|
||||
assert subscribe(client, f"{SITE}/", callback=callback).status_code == 400
|
||||
|
||||
|
||||
def test_unexpected_error_affects_only_that_item(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["good.example"] = echo
|
||||
|
||||
def crash(request):
|
||||
raise RuntimeError("something nobody expected")
|
||||
|
||||
hosts["bad.example"] = crash
|
||||
subscribe(client, f"{SITE}/", callback="https://bad.example/cb") # first in the queue
|
||||
subscribe(client, f"{SITE}/", callback="https://good.example/cb")
|
||||
run_worker(app)
|
||||
assert [s["callback"] for s in subs(app)] == ["https://good.example/cb"]
|
||||
assert intents(app) == []
|
||||
|
||||
|
||||
def test_idn_callback_is_punycoded(app, client, web):
|
||||
assert subscribe(client, f"{SITE}/", callback="https://bücher.example/cb").status_code == 202
|
||||
assert intents(app)[0]["callback"] == "https://xn--bcher-kva.example/cb"
|
||||
|
||||
|
||||
# H3, H4: photos are in what's delivered and in the feeds -----------------------
|
||||
|
||||
def test_new_photo_note_is_delivered_with_its_photo(app, client, web):
|
||||
from starpunk.media import attach_media_to_note, save_media
|
||||
hosts, seen = subscribed(app, client, web)
|
||||
with app.test_request_context():
|
||||
note = make_note(app, "With a photo", has_media=True)
|
||||
assert db(app).execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
media = save_media(jpeg(), "cat.jpg")
|
||||
attach_media_to_note(note.id, [media["id"]], ["A cat"])
|
||||
run_worker(app)
|
||||
(post,) = [r for r in seen if r.method == "POST"]
|
||||
assert media["path"].encode() in post.content
|
||||
|
||||
|
||||
def test_photo_change_refreshes_feed_and_is_delivered(app, client, web):
|
||||
from starpunk.media import attach_media_to_note, save_media
|
||||
with app.test_request_context():
|
||||
note = make_note(app, "Photo later")
|
||||
first = save_media(jpeg("red"), "a.jpg")
|
||||
attach_media_to_note(note.id, [first["id"]], [""])
|
||||
assert first["path"].encode() in client.get("/feed.atom").data # cached now
|
||||
hosts, seen = subscribed(app, client, web)
|
||||
with app.test_request_context():
|
||||
second = save_media(jpeg("blue"), "b.jpg")
|
||||
attach_media_to_note(note.id, [second["id"]], [""])
|
||||
feed = client.get("/feed.atom").data
|
||||
assert second["path"].encode() in feed and first["path"].encode() not in feed
|
||||
run_worker(app)
|
||||
assert second["path"].encode() in [r for r in seen if r.method == "POST"][0].content
|
||||
|
||||
|
||||
# H5: a slow subscriber can't hold the worker ------------------------------------
|
||||
|
||||
def test_slow_callback_is_cut_off():
|
||||
from starpunk import webmention_ssrf as w
|
||||
|
||||
server = socket.socket()
|
||||
server.bind(("127.0.0.1", 0))
|
||||
server.listen(1)
|
||||
port = server.getsockname()[1]
|
||||
|
||||
def trickle():
|
||||
conn, _ = server.accept()
|
||||
conn.recv(65536)
|
||||
try:
|
||||
for b in b"HTTP/1.1 200 OK\r\nX-Slow: " + b"y" * 100:
|
||||
conn.send(bytes([b]))
|
||||
time.sleep(0.3)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
threading.Thread(target=trickle, daemon=True).start()
|
||||
start = time.monotonic()
|
||||
with patch.object(w, "safe_resolve", lambda host: "127.0.0.1"), pytest.raises(httpx.TimeoutException):
|
||||
w.safe_post(f"http://slow.example:{port}/", data={"a": "b"}, timeout=1)
|
||||
assert time.monotonic() - start < 3
|
||||
|
||||
|
||||
# Spec 7: subscriptions stay until their lease ends --------------------------------
|
||||
|
||||
def test_failing_subscriber_is_kept_until_lease_end(app, client, web):
|
||||
hosts, _ = subscribed(app, client, web)
|
||||
hosts["sub.example"] = lambda r: httpx.Response(500)
|
||||
conn = db(app)
|
||||
conn.execute("UPDATE websub_subscriptions SET verified_at = datetime('now', '-10 days')")
|
||||
conn.commit()
|
||||
make_note(app)
|
||||
conn.execute("UPDATE websub_deliveries SET attempts = 5") # last retry
|
||||
conn.commit()
|
||||
run_worker(app)
|
||||
assert conn.execute("SELECT COUNT(*) FROM websub_deliveries").fetchone()[0] == 0
|
||||
assert len(subs(app)) == 1
|
||||
|
||||
|
||||
def test_dns_failure_is_retried(app, client, web, monkeypatch):
|
||||
subscribed(app, client, web)
|
||||
make_note(app)
|
||||
|
||||
def fail(*a, **k):
|
||||
raise socket.gaierror("temporary failure")
|
||||
|
||||
monkeypatch.setattr("starpunk.webmention_ssrf.socket.getaddrinfo", fail)
|
||||
run_worker(app)
|
||||
row = db(app).execute("SELECT attempts, last_error FROM websub_deliveries").fetchone()
|
||||
assert row["attempts"] == 1 and "resolve" in row["last_error"]
|
||||
|
||||
|
||||
def test_render_error_is_retried(app, client, web):
|
||||
subscribed(app, client, web)
|
||||
make_note(app)
|
||||
with patch.object(WebSubWorker, "render_topic", lambda self, t: (500, "text/plain", b"")):
|
||||
run_worker(app)
|
||||
assert db(app).execute("SELECT attempts FROM websub_deliveries").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_gone_topic_ends_subscription_with_denial(app, client, web):
|
||||
hosts, seen = subscribed(app, client, web)
|
||||
make_note(app)
|
||||
with patch.object(WebSubWorker, "render_topic", lambda self, t: (404, "text/html", b"")):
|
||||
run_worker(app)
|
||||
assert subs(app) == []
|
||||
denial = [r for r in seen if r.method == "GET"][0]
|
||||
assert parse_qs(urlparse(str(denial.url)).query)["hub.mode"] == ["denied"]
|
||||
|
||||
|
||||
# M1: limits count pending requests and the callback's address ----------------------
|
||||
|
||||
def test_limits_count_pending_requests(app, client, web):
|
||||
app.config["WEBSUB_MAX_PER_HOST"] = 3
|
||||
codes = [subscribe(client, f"{SITE}/", callback=f"https://s{i}.example/cb").status_code for i in range(5)]
|
||||
assert codes == [202, 202, 202, 429, 429] # all resolve to one address
|
||||
|
||||
|
||||
def test_renewal_is_allowed_at_the_limit(app, client, web):
|
||||
app.config["WEBSUB_MAX_PER_HOST"] = 1
|
||||
subscribed(app, client, web)
|
||||
assert subscribe(client, f"{SITE}/feed.atom", callback="https://sub.example/cb").status_code == 202
|
||||
|
||||
|
||||
# Spec 5.1 details -----------------------------------------------------------------
|
||||
|
||||
def test_unsubscribe_ignores_lease_and_secret(app, client, web):
|
||||
form = {"hub.mode": "unsubscribe", "hub.topic": f"{SITE}/", "hub.callback": "https://sub.example/cb",
|
||||
"hub.lease_seconds": "abc", "hub.secret": ""}
|
||||
assert client.post("/websub", data=form).status_code == 202
|
||||
|
||||
|
||||
def test_unreserved_characters_are_decoded(app, client, web):
|
||||
assert subscribe(client, f"{SITE}/%66eed%2Eatom").status_code == 202
|
||||
assert intents(app)[0]["topic"] == f"{SITE}/feed.atom"
|
||||
|
||||
|
||||
def test_verification_echoes_the_requested_topic(app, client, web):
|
||||
hosts, seen = web
|
||||
hosts["sub.example"] = echo
|
||||
make_note(app, tags=["Python"])
|
||||
subscribe(client, f"{SITE}/tag/Python")
|
||||
run_worker(app)
|
||||
assert parse_qs(urlparse(str(seen[0].url)).query)["hub.topic"] == [f"{SITE}/tag/Python"]
|
||||
assert subs(app)[0]["topic"] == f"{SITE}/tag/python"
|
||||
|
||||
|
||||
def test_content_type_is_case_insensitive(client, web):
|
||||
response = client.post("/websub", data="hub.mode=subscribe&hub.topic=" + SITE + "/&hub.callback=https://sub.example/cb",
|
||||
content_type="Application/X-WWW-Form-Urlencoded")
|
||||
assert response.status_code == 202
|
||||
|
||||
|
||||
def test_empty_secret_is_refused(client, web):
|
||||
assert subscribe(client, f"{SITE}/", **{"hub.secret": ""}).status_code == 400
|
||||
|
||||
|
||||
def test_challenge_must_match(app, client, web):
|
||||
hosts, _ = web
|
||||
|
||||
def padded(request):
|
||||
challenge = parse_qs(urlparse(str(request.url)).query)["hub.challenge"][0]
|
||||
return httpx.Response(200, text=f" {challenge} ")
|
||||
|
||||
hosts["sub.example"] = padded
|
||||
subscribe(client, f"{SITE}/")
|
||||
run_worker(app)
|
||||
assert subs(app) == []
|
||||
|
||||
|
||||
# L1: a newer request isn't lost while an older one is verified ------------------------
|
||||
|
||||
def test_newer_request_survives_older_verification(app, client, web):
|
||||
hosts, _ = web
|
||||
|
||||
def slow_echo(request):
|
||||
# A newer request (new secret) arrives while this one is being verified
|
||||
subscribe(client, f"{SITE}/", **{"hub.secret": "new"})
|
||||
return echo(request)
|
||||
|
||||
hosts["sub.example"] = slow_echo
|
||||
subscribe(client, f"{SITE}/", **{"hub.secret": "old"})
|
||||
run_worker(app)
|
||||
pending = intents(app)
|
||||
assert len(pending) == 1 and pending[0]["secret"] == "new"
|
||||
|
||||
|
||||
# L2: fewer renders and pushes ------------------------------------------------------
|
||||
|
||||
def test_topic_rendered_once_per_pass(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["a.example"] = hosts["b.example"] = echo
|
||||
for cb in ("https://a.example/cb", "https://b.example/cb"):
|
||||
subscribe(client, f"{SITE}/feed.atom", callback=cb)
|
||||
run_worker(app)
|
||||
make_note(app)
|
||||
calls = []
|
||||
original = WebSubWorker.render_topic
|
||||
|
||||
def counting(self, topic):
|
||||
calls.append(topic)
|
||||
return original(self, topic)
|
||||
|
||||
with patch.object(WebSubWorker, "render_topic", counting):
|
||||
run_worker(app)
|
||||
assert calls == [f"{SITE}/feed.atom"]
|
||||
|
||||
|
||||
def test_like_only_updates_likes_page(app, client, web):
|
||||
hosts, _ = web
|
||||
hosts["sub.example"] = echo
|
||||
for topic in (f"{SITE}/", f"{SITE}/likes"):
|
||||
subscribe(client, topic)
|
||||
run_worker(app)
|
||||
make_note(app, "", like_of="https://elsewhere.example/post")
|
||||
topics = [r[0] for r in db(app).execute("SELECT topic FROM websub_deliveries")]
|
||||
assert topics == [f"{SITE}/likes"]
|
||||
|
||||
|
||||
# L5: feeds ----------------------------------------------------------------------
|
||||
|
||||
def test_etag_is_stable_across_processes(app, client):
|
||||
from starpunk.feeds import get_cache
|
||||
make_note(app)
|
||||
first = client.get("/feed.rss").headers["ETag"]
|
||||
get_cache().invalidate() # as if another process rendered it
|
||||
time.sleep(1.1) # a different build time
|
||||
assert client.get("/feed.rss").headers["ETag"] == first
|
||||
|
||||
|
||||
def test_negotiated_feed_varies_on_accept(client):
|
||||
assert "Accept" in client.get("/feed", headers={"Accept": "application/atom+xml"}).headers.get("Vary", "")
|
||||
Reference in new issue
Block a user